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1  Executive  Summary 

This  report  summarizes  the  work  performed  during  this  Phase  II  Small  Business  Innovation  Research 
(SBIR)  project.  All  objectives  stated  in  the  Phase  II  proposal  were  accomplished.  Furthermore,  Klein 
Associates,  Inc,  the  primary  sub-contractor  on  the  effort,  along  with  the  SHAI  team  engaged  in 
significant  interaction  with  LSOs,  which  was,  and  will  be,  incredibly  important  for  this  effort. 


2  Problem  Statement 

The  Aircraft  Carrier,  or  CV,  landing  environment  is  an  extremely  complex  one.  In  addition  to  operating 
what  may  be  termed  as  an  extremely  busy  airport,  CV  landing  operations  are  affected  by  a  number  of 
variables  not  associated  with  a  normal  aerodrome.  Of  these,  the  most  critical  are  fleet  tactical 
considerations,  flight  deck  space  constraints,  CV  maneuvering  space  (sea  room),  flight  deck  motion 
(pitch  and  roll),  continuous  mechanical  preparations,  resetting  arresting  gear  and  optical  landing  system 
between  each  landing,  airborne  aircraft  fuel  status  and  management  of  aerial  refueling  assets,  aircraft 
ordinance,  minimal  use  of  navigation,  communications  and  radar  emissions  as  in  EMCON  operations 
and,  above  all,  time  constraints. 

Safe  and  efficient  control  of  this  environment  requires  following  a  strict  chain  of  command  and  adhering 
to  a  set  of  standard  operating  procedures.  The  chain  of  command  follows  from  the  CV  Captain,  through 
the  Air  Operations  Officer  (Airops),  below  the  flight  deck,  and  the  Air  Officer  (Air  Boss),  in  the  tower, 
to  the  Landing  Signal  Officer  (LSO),  stationed  at  the  stem  of  the  ship  next  to  the  landing  area.  The 
Captain  is  ultimately  responsible  for  the  entire  operation  of  his  ship.  The  Airops  Officer  is  responsible 
for  aircraft  outside  a  five  mile  radius  to  a  distance  of  twenty  miles  from  the  ship,  as  well  as  managing 
airborne  fuel/aerial  refueling  and  aircraft  status  and  providing  surveillance  and  precision  radar  guidance 
to  the  pilots  for  both  night  and  low  visibility  approaches  and  landings.  The  Air  Boss  is  responsible  for 
aircraft  within  five  nautical  miles  as  well  as  all  flight  deck  preparations,  aircraft  handling  bn  the  flight 
deck,  and  final  landing  clearance.  The  LSO  is  responsible  for  the  aircraft’s  final  approach  and  landing. 

During  the  last  60  seconds,  the  cognitive  demands,  namely  the  critical  decisions  and  judgments,  increase 
quickly  until  a  decision  to  wave  off,  or  not,  is  made.  Day  landings  with  good  weather  are  ideal 
conditions,  but  unfortunately  not  all  days,  or  nights,  are  like  that.  Often  times  the  ship  is  heaving  10ft. 
up  and  10  ft.  down,  making  a  20  ft.  displacement  from  a  level  deck.  In  addition,  it  is  often  difficult  to 
see  the  aircraft  approach  during  night  operations,  and  impossible  to  see  during  stormy  conditions  at 
night.  The  LSOs  must  rely  on  auditory  cues  and  the  equipment  at  the  LSO  station  to  assist  their  decision 
making.  For  this  project,  we  were  tasked  to  design  a  decision  support  tool  that  will  assist  LSO  decision 
making  and  hopefully  increase  the  amount  of  time  to  make  a  wave-off  decision,  which  is  usually  about 
0.5-4  seconds.  The  focus  of  the  decision  support  tool  was  to  provide  pilot  trending  information  along 
with  key  oscillation  deviations  so  that  the  LSO  could  improve  both  safety  and  efficiency  of  recoveries. 
We  feel  that  we  have  done  this  with  our  current  interface,  and  have  taken  the  assignment  one  step  further 
in  providing  predictions  for  aircraft  and  deck  position,  two  key  oscillations  during  flight  operations.  This 
report  will  describe  the  LSO  environment,  the  approach  we  took  to  investigating  pilot  trending  and  key 
oscillations,  and  the  development  of  a  decision  support  interface  to  be  implemented  in  the  VISUAL 
system,  a  larger  information  system  scheduled  to  be  included  on  the  LSO  platform. 
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2.1  Day  Operations 

Daytime  landing  operations  are  referred  to  as  Case  I  recoveries.  With  Case  I  recoveries,  the  aircraft  fly 
by  the  starboard  side  of  the  ship  (downwind)  and  perform  a  break  once  they  are  past  the  bow.  The 
aircraft  continues  the  turn  to  fly  upwind  at  about  1  mile  off  the  port  side  into  a  final,  gradual  180  degree 
turn  and  arrives  at  a  point  1/2  to  3/4  of  a  mile  astern  the  ship.  The  landing  area  is  then  prepared  for 
recoveiy,  which  is  normally  within  a  45  to  60  second  separation  interval  from  the  aircraft  ahead.  The 
Senior  LSO  or  one  of  the  two  Airwing  Staff  LSOs  “wave”  the  approaching  aircraft  from  the  LSO 
Platform. 

(See  Picture  1  attached  at  the  end  of  the  document). 

Several  other  individuals  assist  these  LSOs;  some  are  LSO's  and  some  are  enlisted  personnel.  Normally 
the  most  senior  LSO  will  be  the  “backup”  or  supervising  LSO  while  a  qualified,  but  more  junior 
individual  will  be  the  “controlling”  LSO.  A  third  individual  will  copy  shorthand  comments  into  the 
LSO's  Grade  Log  for  use  after  the  recovery  in  debriefing  each  pilot  regarding  his  approach. 

Occasionally,  other  less  experienced  LSO's  will  observe  the  recovery  for  training.  Information  available 
to  the  LSO  on  the  platform  is  provided  in  console  displays  including:  Wind-over-the  deck;  Optical 
Landing  System  Status  and  lamp  intensity  indicators  and  controls;  Clear  Deck/Foul  Deck  (green  light/red 
light)  indicator;  deck  motion  indicators;  and  a  CRT  display  from  a  fixed  centerline  video  camera  view  up 
the  glidepath  with  stabilized  crosshairs.  This  final  source  of  information  is  the  Pilot  Landing  Aid 
Television  (PLAT)  from  which  a  videotape  is  recorded  for  later  reference  as  well  as  mishap 
investigation.  The  LSO's  communications  resources  consist  of  two  radio  sets  (under  his  control),  access 
to  two  radar  controller  radio  circuits,  a  ship's  telephone,  several  sound-powered  phone  circuits  and  one  or 
two  enlisted  deck  personnel  in  constant  communication  with  other  critical  recovery  operations 
workstation  operators.  The  Captain  and  Air  Boss  can  also  communicate  with  the  LSOs  via  their 
individual  public  address  systems  from  the  ship's  tower. 

The  deck  preparation  includes  stowing  loose  equipment,  moving  personnel  from  their  launch  to  their 
recovery  positions,  moving  aircraft  from  the  landing  area,  and  retracting  the  arresting  cable. 
Simultaneously,  the  Optical  Landing  System  is  reset  for  the  next  type  of  aircraft  approaching,  and  an 
enlistee  on  the  LSO  platform  is  also  checking  that  the  approaching  aircraft  is  in  the  proper  configuration 
for  landing  (landing  gear  locked  down,  tail-hook  down  and  flaps/slats  extended).  As  the  aircraft 
approaches,  the  LSO  assesses  the  pilot's  response  to  deviations  from  on-speed,  centerline,  and  glideslope. 
None  of  these  factors  remain  constant  as  the  landing  area  is  angled  and  constantly  moving  to  the  right  of 
the  aircraft's  flight  path  as  the  ship  moves  forward.  The  aerodynamics  of  the  flight  path  is  affected  by 
the  wind  over  the  deck  as  it  flows  down  off  the  stem  then  rises  off  the  water  1/4  to  1/2  mile  behind  the 
ship  at  varying  intensities.  This  effect  is  associated  with  a  variance  of  aircraft  airspeed  as  the  pilot 
constantly  corrects  for  these  and  his  own  induced  deviations.  At  a  point  in  the  aircraft's  approach, 
usually  within  0.5  -  4  seconds  of  recovery,  the  LSO  must  then  determine  whether  the  aircraft  position 
and  speed  is  stable  and  safe  enough  to  complete  a  landing  to  a  touchdown  area  approximately  20  feet 
wide  by  200  feet  long,  or  signal  the  pilot  to  execute  a  wave-off.  If  the  deck  is  still  being  prepared  for  the 
landing,  the  LSO  must  judge  whether  or  not  the  deck  can  be  cleared  in  time  for  a  safe  recovery.  At  the 
completion  of  a  wave-off  or  landing  the  LSO  will  again  begin  assessing  the  next  aircraft  while  verbally 
evaluating  the  last  approach  to  his  assistant. 
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2.2  Night  Operations 

Night  operations  fall  into  the  Case  II  or  Case  III  recovery  classification.  Case  11  recoveries  follow  the 
same  approach  pattern  as  Case  I  (daytime)  operations,  while  Case  III  recoveries  have  the  aircraft 
marshalling  approximately  20  miles  out  and  they  come  straight  in  to  the  carrier.  Case  III  recoveries 
encompass  situations  where  visibility  is  very  low  and/or  weather  conditions  are  poor.  These  are  the  most 
difficult  recoveries.  In  all  three  types  of  recoveries,  the  approach  patterns  are  identical  from 
approximately  %  of  a  mile  from  the  ramp  on  in. 

In  Case  III  recoveries,  the  marshalling  area  is,  as  mentioned,  20  miles  out.  The  aircraft  then  follow  an 
instrument  approach  procedure  to  arrive  at  a  point  10  miles  behind  the  ship,  then  receive  either  verbal 
Precision  Approach  Radar  guidance,  a  precise  Instrument  Landing  System  display  in  the  cockpit,  or  are 
automatically  controlled  to  touchdown  by  coupling  the  aircraft's  auto  pilot  to  the  ship's  Automatic 
Carrier  Landing  System.  These  approaches  are  generally  more  stable  at  the  start  due  to  the  long  straight- 
in  flight  path  versus  a  turning  approach  to  a  short  wings-level  final  as  in  Case  I  and  II.  The  LSO  is  also 
presented  with  repeaters  of  the  aircraft's  performance  via  the  information  directly  accessed  from  the 
precision  radar.  This  is  presented  on  a  collector  lens  directly  in  front  of  the  LSO  in  the  form  of  a  Heads 
Up  Display  (HUD).  The  LSO  can  use  this  information  to  anticipate  errors  from  these  visual  and  aural 
cues.  The  HUD  is  of  greatest  use  when  meteorological  conditions  restrict  the  LSOs  normal  ability  to 
watch  the  aircraft  approach  visually.  It  is  under  these  visually  restricted  conditions  (darkness  on  the 
flight  deck,  darkness  looking  astern  the  ship,  and  the  associated  loss  of  depth  perception  at  night,  etc.), 
that  the  LSOs  face  their  greatest  challenges. 

2.3  LSO  Chaiienges 

The  Landing  Signal  Officer  faces  many  challenges.  Although  LSOs  primary  concern  remains  safety,  the 
ship  is  under  significant  pressure  to  maintain  an  extremely  rapid  recovery  rate.  The  average  recovery 
rates  necessitate  small  intervals  (45  to  60  seconds)  between  each  landing,  and  there  is  constant  pressure 
to  not  wave-off  unless  absolutely  necessary.  Additionally,  due  to  EMCON  constraints,  the  LSO 
minimizes  use  of  the  radio.  Consequently,  LSOs  have  little  time  to  make  the  wave-off  decision,  are 
often  forced  to  wait  until  the  last  possible  second  to  make  the  final  decision,  and  often  may  not  have  a 
good  definition  of  what  the  last  second  is.  Many  of  the  issues  an  LSO  must  address  are  juggled  between 
a  series  of  trade-offs:  safety  vs.  effectiveness;  wave-off  a  bad  pass  vs.  getting  the  pilot  on  board,  etc. 

During  day  operations  in  clear  weather  conditions,  the  approach  radar  is  not  normally  used  and  therefore 
much  of  the  information  that  would  be  available  from  this  equipment  (e.g.,  speed,  actual  rate-of  descent, 
distance,  etc.)  is  not  directly  available.  During  these  approaches,  the  LSOs  visually  ascertain  aircraft 
attitude,  from  which  air  speed  can  be  inferred.  As  the  LSO  visually  monitors  the  aircraft  for  proper 
glideslope  and  line-up,  current  throttle  setting  can  also  be  inferred  by  listening  to  the  engine.  The  LSO  is 
also  adept  at  predicting  what  the  pilots  next  move  might  be  based  on  the  dynamics  of  the  flight  path. 

This  is  an  important  parameter  because  to  reject  the  landing,  the  pilot  must  go  to  full-throttle,  which  may 
take  up  to  6.5  seconds  to  take  effect,  depending  on  the  current  throttle  settings  and  aircraft  type.  This  is  a 
considerable  period  of  time,  given  the  split-second  decision-making  performed  by  the  LSO. 

In  addition  to  recognizing  particular  aircraft  and  pilot  model  (different  for  each  pilot  and  aircraft  type), 
the  LSO  must  also  consider  status  of  the  deck  condition  and  crew,  specifically  whether  the  deck  is  clear 
and  crew  is  ready  for  recovery.  If  the  deck  is  foul,  the  Air  Boss  will  only  notify  the  LSO  if  it  is 
absolutely  sure  that  the  deck  will  not  be  ready  in  time  and  will  call  a  no  chance  wave-off.  Normally  the 
LSO  will  wait  until  the  last  possible  moment  to  wave  the  aircraft  off,  since  the  deck  may  be  ready  just  at 
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glideslope. 

In  addition  to  having  little  precise  and  having  various  pilot 

extremely  high  time  pressure  ®  ®LSC)  must  estimate  the  wave-off  window  as  it  changes 
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safety  considerations.  Otherwise,  LSOs  have  learned  that  having  too  much  of  an  LSO 

matters  worse.  Being  active  Navy  pilots  ,  learned  that  it  complicates 

in  the  cockpit  can  be  very  confusing.  A  o  §  instructions  along  more  than  one  dimension  at  a  time 
things  when  they  try  to  give  pilots  simultaneously^  They  recognize  that  they  have  to 

(e.g.,  passing  on  line-up  and  glide  slope  co  critical  corrections  first, 

prioritize  the  correction  information  they  pass  on,  giving  the  pilot  me 
followed  later  by  additional  less  critical  corrections. 

ti^HifWntatniffht  Night  recoveries  are  considered  much  more 

The  LSOs  and  pilots  situation  is  greatly  different  S  '  ®  ^  approach  radar,  which  gives 

dangerous  than  daylight  operations.  “"f„n„ation  such  that  deviations  from  glideslope  and 

them  more  accurate  distance,  azimuft,  and  '>'“""8  H  on  the  LSO  Heads  Up 

line-up  can  be  readily  calcukted  and  displayed  jn  the  trends  that  the 

Display.  Unfortunately,  the  accuracy  aLrate  deviation 

LSO  is  monitoring  when  compared  to  ^^^s  visual  perceptions  of 

^J^ShtdT— and  glideslope  conhol  and  is  therefore  challenged  wid, 
integrating  multiple  information  resources. 

j  f»vprv  nilot  is  graded.  So,  in  addition  to  controlling 

Sp  LSOs  a^ra  also  yelling  out  their  observations  for  fte  Logging 

LSO  to  record  for  later  use  in  the  pilot  debrief. 

L  goaUf  rpS,  in  addrtion  to 

a  combination  of  decision-centered  techniques  from  the  field  ot  co^ra  brings  expertise  in 

system  techniques  from  the  field  of  Artificial  In  elhgen  e  ^ 

cognitive  psychology  and  decision-centered  e  and  they  include:  decision-centered 

LogS’Neural  Networks,  and  Neural  Networks  Based  Fuzzy  Inference  Syste  . 
wonting  with  Landing  Signal  Officers  to  — «ow  ftey 
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learn  the  overall  task  of  recovering  aircraft  aboard  U.S.  aircraft  carriers.  It  was  a  fairly  broad  approach 
that  did  not  restrict  itself  to  only  the  LSOs  on  the  LSO  platform,  but  also  took  into  consideration  the  roles 
of  enlisted  personnel  on  the  platform  as  well  as  those  individuals  in  the  tower  and  air  operations  (Air 
Ops).  The  goal  of  the  Phase  I  research  was  to  investigate  the  feasibility  and  usefulness  of  combining 
these  cognitive  and  AI  approaches.  The  Phase  I  resulted  in  display  recommendations  for  the  Controlling 
LSO  (the  individual  who  actually  controls  the  aircraft  through  its  final  approach  to  the  recovery). 

For  the  Phase  II,  we  concentrated  specifically  on  pilot  trending  and  oscillation  considerations.  Our 
objective  in  this  Phase  II  effort  was  to  identify  the  critical  pilot  trends  and  oscillations  that  an  LSO  must 
contend  with,  and  to  build  an  advanced  decision-support  interface  that  supports  the  split-second 
decisions  and  perceptual  workload  of  the  LSO.  We  identified  three  dynamic  aspects  of  the  approach 
oscillations:  the  aircraft,  the  deck,  and  the  individual  pilot  (preferences,  habits,  and  trends). 

Our  work  is  consistently  guided  by  an  approach  to  system  design  we  have  termed  decision-centered.  In 
the  sections  that  follow,  we  will  provide  descriptions  of  decision-centered  design  and  our  approach  to 
CTA  knowledge  elicitation  and  representation.  Following  that,  we  will  introduce  multiple  AI  techniques 
that  were  used  to  assimilate  data  collected  from  CTA  knowledge  elicitation.  We  turn  then  to  a 
discussion  of  results  of  data  collection  as  they  relate  to  the  problems  and  issues  that  surround 
development  of  a  sturdy  and  flexible  interface  for  the  LSO  operator. 

3.1  Decision-Centered  Design 

A  decision-centered  approach  (Kaempf  &  Miller,  1993;  Klein,  1993;  Klein,  Kaempf,  Wolf,  Thordsen, 
Miller,  1997)  to  design  involves  using  Cognitive  Task  Analysis  (CTA)  to  identify  the  critical  decisions, 
judgments,  and  cognitive  elements  of  the  task  and  then  applying  this  information  to  any  of  a  number  of 
purposes.  A  decision-centered  approach  is  best  understood  when  presented  in  contrast  to  data-centered 
and  system-centered  approaches.  The  distinction  is  an  important  one  for  understanding  consequences  for 
design  and  operator  performance,  and  is  described  below. 

Many  approaches  to  system  design  have  been  driven  by  the  information-processing  power  of  emerging 
new  technologies.  The  capabilities  of  these  high  power  systems  permit  access  to  vast  amounts  of  raw 
data.  Any  or  all  of  these  data  may  be  made  available  to,  and  sometimes  even  imposed  on,  the 
users/operators.  The  display  and  control  design  is  data-centered  and  technology  driven.  This  is  not 
surprising  given  the  great  temptation,  in  the  face  of  extraordinary  memory  capacity  and  operating  power, 
to  provide  as  much  information  to  the  individual  as  is  technologically  possible.  The  problem  with  a  data- 
centered  approach  is  that  it  does  not  take  into  account  the  shifting  contexts  in  which  many  users  function, 
it  totally  ignores  what  the  user  needs,  when  he  or  she  needs  it,  or  how  it  should  be  represented.  We  have 
seen  many  situations  where  individuals  (e.g.,  pilots  and  other  operators  of  complex  systems)  begin  by 
turning  off  various  support  and  warning  systems  because  the  operators  say  they  are  so  distracting.  Vast 
amounts  of  generic  data  can  interfere,  can  result  in  information  overload,  and  can  force  the  operator  to 
use  valuable  time  sorting  through  data  that  may  be  important  in  other  situations,  in  order  to  find  the  one 
or  two  pieces  that  are  important  in  this  situation.  More  is  not  always  better.  In  fact,  more  is  dangerous. 

A  second  approach  can  be  termed  system-centered.  The  human  factors  community  has  recognized  the 
difficulties  of  design  that  is  technology  driven.  They  understand  that  the  information  requirements  of  the 
user  are  of  utmost  importance  if  a  complex  system,  which  includes  both  the  individual  and  the 
equipment,  is  to  perform  effectively.  One  approach  to  this  has  been  system-centered  designs,  where  data 
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are  organized  and  presented  within  the  context  of  the  various  electronic  or  mechanical  sub-systems.  For 
example,  on  an  aircraft  these  systems  might  include  fuel,  weapons  status,  hydraulics,  and  navigation.  For 
the  LSOs  these  might  include  the  aircraft,  the  deck,  the  weather,  etc.  Overall,  the  system-centered 
approach  has  allowed  marked  improvements  in  interface  design.  It  presents  data  organized  in  a  way  that 
helps  the  user  understand  the  status  of  various  systems  and  equipment.  In  the  cockpit  example, 
information  for  aircraft  control,  navigation,  fuel  management,  ordnance  management,  and  tactical  and 
mission  data  are  all  available  to  the  pilot  in  various  forms  and  displays.  However,  the  system-centered 
approach  does  not  recognize  that  the  data  it  provides  generally  plays  a  supporting  role  to  the  cognitive 
processes  that  the  users  require  in  order  to  achieve  their  missions.  In  other  words,  while  the  system 
provides  a  variety  of  data  elements,  it  is  left  to  the  user  to  synthesize  the  data  to  answer  a  particular 
question,  or  to  fit  data  to  the  needs  of  the  current  situation.  For  example,  a  system  display  that  shows  a 
pilot  the  status  of  the  fuel  system  is  very  helpful,  however,  this  information  is  usually  only  a  sub-part  of 
the  overall  information  the  pilot  may  need.  More  often  than  not,  fuel  status  is  related  to  other  factors 
such  as  distance  and  time,  and  in  the  aviation  domain,  wind  and  speed  emerge  as  important  factors  as 
well.  The  integration  of  these  factors  would  be  more  useful,  perhaps,  than  just  a  fuel  status  display. 

It  is  our  view  that  for  the  user  to  think  and  act  effectively,  that  data  presentation  has  to  take  into  account 
the  context  of  the  individuals  decision  making,  rather  than  being  data-  or  system-centered.  In  effect,  it 
needs  to  be  decision-centered.  To  achieve  this,  the  information  must  be  presented  in  a  functionally 
meaningful  way  the  information  must  be  framed  by  the  nature  of  the  critical  decisions  and  judgments 
(i.e.,  decision  requirements)  within  a  particular  context  or  situation,  and  made  available  to  system  users 
in  ways  that  support  thinking  and  action. 

A  decision-centered  approach,  as  the  name  implies,  anchors  the  design  around  the  decisions  that  will 
confront  the  user  who  is  involved  with  the  systems  tasks.  A  decision-centered  approach  can  be  viewed  as 
a  variation  of  Cognitive  Systems  Engineering,  one  in  which  decision  requirements  (the  most  critical  and 
difficult  decisions  and  judgments)  provide  the  foundation  for  the  generation  of  the  design  principles  and 
recommendations.  Decision-centered  design  is  beneficial  in  any  domain  that  involves  interaction  of 
human  and  smart  machines  to  accomplish  complex  tasks.  However,  in  domains  that  involve  extreme 
time  pressure  and  risk,  as  in  the  LSO  domain,  decision-centered  design  is  critically  important  to  optimal 
performance  and  avoidance  of  mistakes. 


3.2  Cognitive  Task  Anaiysis 

Our  approach  to  decision-centered  design  is  grounded  in  the  use  of  Cognitive  Task  Analysis  (CTA)  tools 
and  techniques  that  are  associated  with  critical  event  and  critical  decision  methodologies.  The  key 
element  of  these  approaches  is  that  they  derive  their  data  from  a  combination  of  observations  of  and 
interviews  about  real-world  decisions  and  events.  For  this  project,  the  CTA  helped  us  identify  and 
document  cognitive  elements  (i.e.,  critical  decisions  and  judgments)  of  the  LSO  task,  so  that  we  could 
incorporate  these  critical  elements  into  the  design  of  a  flexible  and  high-powered  interface. 

CTA  as  conducted  by  Klein  Associates  researchers  is  an  in-depth  examination  of  an  individuals  expertise 
in  the  context  of  his  or  her  job  the  cues  and  patterns  of  cues,  strategies,  challenges,  discriminations, 
assessments,  expectancies,  goals,  and  ability  to  detect  and  anticipate  problems.  These  methods  are  in 
contrast  to  other  more  generic  or  abstract  CTA  methods  derived  from  a  task  analysis  approach.  Rather 
than  decompose  the  task  into  subcomponents,  the  intent  of  our  CTA  methods  is  to  get  inside  the  head  of 
the  expert  and  understand  the  rich  cognitive  elements  that  are  difficult  to  articulate  For  The  CTA  tools 
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we  used  to  uncover  these  issues  include  the  Critical  Decision  Method  (CDM)  and  the  Knowledge  Audit, 
and  are  described  below. 


3.2.1  Critical  Decision  Method 

Klein  Associates  most  frequently  used  research  tool,  the  Critical  Decision  Method  (CDM),  has  been  used 
in  dozens  of  studies  of  decision  making  and  problem  solving.  CDM  interviews  are  based  on  Flanagan's 
(1954)  critical  incident  technique  and  are  organized  around  an  initial,  unstructured  account  of  a  specific 
incident.  The  incident  account  is  generated  by  the  interviewee  in  response  to  a  specific  open-ended 
question  posed  by  the  interviewers,  and  it  provides  the  structure  for  the  interview  that  follows.  By 
requesting  accounts  of  a  certain  type  of  event,  and  structuring  the  interview  around  that  account, 
potential  interviewer  biases  are  minimized.  Once  the  report  of  the  incident  has  been  completed,  the  CDM 
interviewer  leads  the  participant  back  over  his  or  her  incident  account  several  times,  using  probes 
designed  to  focus  attention  on  particular  aspects  of  the  incident  and  solicit  information  about  them. 
Solicited  information  depends  on  the  purpose  of  the  study,  but  might  include  presence  or  absence  of 
salient  cues  and  the  nature  of  those  cues,  assessment  of  the  situation  and  the  basis  of  that  assessment, 
expectations  about  how  the  situation  might  evolve,  goals  considered,  challenges  faced,  and  options 
evaluated  and  chosen.  And  because  information  is  elicited  specific  to  a  particular  decision  and  incident, 
the  context  in  which  the  decision  maker  is  operating  in  remains  intact  and  becomes  part  of  the  data 
record. 

CDM  has  been  highly  successful  in  eliciting  perceptual  cues  and  details  of  judgment  and  decision 
strategies  that  are  generally  not  captured  with  traditional  reporting  methods,  and  has  been  demonstrated 
to  yield  information  richer  in  variety,  specificity,  and  quantity  than  is  typically  available  in  experts' 
unstructured  verbal  reports  (Crandall,  1989).  The  information  obtained  via  these  methods  is  concrete  and 
specific,  reflects  the  point  of  view  of  the  decision  maker,  and  is  grounded  in  experience.  Detailed 
descriptions  of  CDM  and  other  work  surrounding  it  can  be  found  in  Klein  et  al.  (1989)  and  Hoffman  et 
al.(1998). 


3.2.2  Knowledge  Audit 

Another  CTA  tool  is  the  Knowledge  Audit,  developed  under  a  contract  with  the  Navy  Personnel 
Research  and  Development  Center  (Militello  et  al.,  1997).  The  objective  was  to  develop  a  streamlined  set 
of  CTA  tools,  which  could  be  used  effectively  by  people  outside  of  the  cognitive  research  community. 
The  Knowledge  Audit  focuses  on  the  categories  of  knowledge  and  skills  that  distinguish  experts  from 
others.  These  categories  include  metacognition,  mental  models,  perceptual  cues  and  patterns,  analogues, 
and  declarative  knowledge.  The  Knowledge  Audit  provides  an  efficient  method  for  surveying  the  various 
aspects  of  expertise.  The  method  does  not  attempt  to  find  whether  each  component  of  expertise  is  present 
for  a  given  task.  Rather,  it  employs  a  set  of  specific  probes  designed  to  describe  the  t3rpe  of  knowledge  or 
skill  and  to  elicit  examples  of  each  based  on  actual  experiences.  The  primary  strength  of  the  Knowledge 
Audit  is  that  it  enables  us  to  survey  rapidly  the  nature  and  breadth  of  skills  involved  in  expertise 
in  a  given  domain. 

The  next  section  will  describe  the  Artificial  Intelligence  (AT)  methods  that  were  used,  in  conjunction 
with  data  derived  from  the  CTA  methods,  to  build  an  advanced  decision-support  tool  for  the  Landing 
Signal  Officer  workstation.  CTA  data  was  a  critical  ingredient  in  building  an  intelligent  system,  and  a 
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more  detailed  account  of  how  the  CTA  data  fit  into  AI  models  of  design  is  described  in  later  sections. 
Next  is  a  description  of  AI  methods  utilized  in  this  project. 

3.3  Case-Based  Reasoning 

Case-Based  Reasoning  (CBR)  reasoning  is  a  knowledge  representation  and  control  methodology  based 
upon  previous  experiences  and  patterns  of  previous  experiences.  These  previous  experiences  (previous 
carrier  landings),  or  “cases”  of  domain-specific  knowledge  and  action,  are  used  in  comparison  with  new 
situations  or  problems.  These  past  methods  of  solution  provide  expertise  for  use  in  new  situations  or 
problems. 

Much  of  the  research  in  Case-Based  Reasoning  is  directed  toward  retrieving  similar  cases  and 
determining  useful  definitions  of  similarity.  It  was  found  to  be  useful  in  retrieving  similar  or  relevant 
past  approaches.  For  a  pilot  trending  system,  the  cases  are  simply  previous  examples  of  carrier  landings, 
including  all  information  available  from  the  ship’s  systems,  from  which  inferences  and  comparisons  can 
be  made  using  CBR. 

Because  CBR  is  based  on  the  ways  humans  think,  it  is  a  very  natural  way  to  support  the  human  decision¬ 
making  process.  For  example,  the  LSOs  all  described  how  important  it  was  to  have  seen  several 
previous  approaches  made  by  the  incoming  pilot.  The  CBR  system  aids  this  process  by  retrieving 
similar  approaches,  in  case  this  particular  LSO  does  not  have  much  experience  with  this  particular  pilot, 
for  these  conditions. 

By  retrieving  a  set  of  very  similar,  very  relevant  cases,  the  CBR  system  helps  the  LSO  make  qualitative 
assessments  of  the  current  approach  based  on  the  particular  pilot’s  tendencies  and  trends.  This 
assessment  is  based  on  the  statistics  of  the  retrieved  similar  cases,  weighted  based  on  their  degree  of 
similarity  to  the  current  situation. 

3.4  Fuzzy  Logic 

Zadeh  introduced  the  concept  of  fuzzy  logic  in  1965.  Since  then,  fuzzy  logic  has  advanced  in  a  wide 
variety  of  disciplines  such  as  control  theory,  topology,  linguistics,  optimization,  and  category  theory. 
Unlike  a  crisp  set,  a  fuzzy  set  allows  partial  membership.  Fuzzy  logic  is  a  generalization  of  the 
traditional  TRUE/FALSE  bi-level  logic,  one  that  allows  for  non-sharp  transition,  representing  a  region  of 
partial  truth,  between  absolute  true  and  absolute  false.  For  example,  although  the  assertion  that  an 
individual  is  male  is  either  true  or  false  (and  is  therefore  crisp),  the  assertion  that  an  individual  is  lean  is 
not  so  clear-cut.  Figure  1  demonstrates  how  the  fuzzy  sets  may  be  used  to  capture  this  concept.  A 
person  with  a  body  fat  percentage  of  1 6.5  has  membership  values  of  0. 1 2  and  0.43  in  the  "lean"  and 
"moderately  overweight"  fuzzy  sets,  respectively. 

The  basic  architecture  of  a  fuzzy  logic  data  analysis  system  is  illustrated  in  Figure  2.  The  numerical 
input  data  is  codified  through  the  fuzzifier  into  the  equivalent  linguistic  parameters  (such  as  lean, 
moderately  overweight,  and  obese),  with  associated  membership  function  values.  The  inference  engine 
uses  the  knowledge  in  a  particular  representation  to  derive  some  expert  conclusion  or  offer  expert  advice. 
It  includes  the  system's  general  problem-solving  knowledge.  Various  rules  in  the  knowledge  base  and 
decision-making  logic  are  invoked  and  recover  the  decision  actions  with  different  degrees  of  emphasis 
depending  on  their  respective  membership  values. 
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Figure  1.  Fuzzy  Membership  Functions 


The  final  stage  in  the  fuzzy  logic  data  processor  aggregates  all  the  inferred  fuzzy  data  and  produces  an 
appropriate  conclusion  or  classification  of  the  system's  input.  If  the  system's  output  needs  to  be  in  non- 
fuzzy  numerical  format,  it  is  the  responsibility  of  the  defuzzification  module  to  convert  fuzzy  data  to 
numerical  from. 


Figure  2.  General  architecture  of  fuzzy  logic  data  analysis  system 

Fuzzy  logic  was  found  to  be  useful  in  conjunction  with  case -based  reasoning  to  determine  similar 
landings.  In  addition,  fuzzy  logic  was  found  beneficial,  when  combined  in  a  neural  network  type  system, 
in  the  prediction  of  future  aircraft  locations. 

3.5  Neural  Networks 

Neural  networks  are  an  approach  to  machine  learning  which  developed  out  of  attempts  to  model  the 
processing  that  occurs  within  the  neurons  of  the  brain.  By  using  simple  processing  units  (neurons), 
organized  in  a  layered  and  highly  parallel  architecture,  it  is  possible  to  perform  arbitrarily  complex 
calculations.  Learning  is  achieved  through  repeated  minor  modifications  to  selected  neurons,  which 
results  in  a  very  powerful  classification  system.  Neural  network  software  is  used  to  recognize,  and  also 
to  run  at  desired  conditions.  Applications  include  handwriting  recognition,  fingerprint  identification, 
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control  of  chemical  processes,  speech  recognition,  credit  analysis,  scientific  analysis  of  data,  and  in 
neurophysiological  research.  Neural  networks  are  also  referred  to  as  neural  nets,  connectionism,  and 
parallel  associative  memory. 

Neural  networks  techniques  were  utilized  in  conjunction  with  fuzzy  logic  techniques  to  create  a  neural 
network  based  fuzzy  inference  system  for  learning  predictions  for  future  aircraft/pilot  locations. 

3. 6  Neural  Network  Based  Fuzzy  Inference  System 


output 

vector 


A  neural  network  based  fuzzy  inference  system  (Figure  4)  is  a  multi-layer  network  in  which  each  node 
performs  a  particular  function  (e.g.,  a  fuzzy  function)  on  incoming  signals  (as  well  as  a  set  of  parameters 
pertaining  to  the  node).  The  nature  of  the  node  function  may  vary  from  node  to  node,  and  the  choice  of 
each  node  function  depends  on  the  overall  input-output  function  which  the  neural  network  is  required  to 
carry  out.  A  neural  network  has  two  types  of  nodes:  an  adaptive  node  (represented  by  a  square  in  Figure 
4)  has  parameters  that  may  be  updated  by  a  learning  algorithm,  while  a  fixed  node  (represented  by  a 
circle)  has  none.  A  neural  network-based  fuzzy  inference  system  is  comprised  of  several  layers  of  nodes, 
as  illustrated  in  Figure  4.  The  node  function  of  each  node  in  the  premise  layer  of  nodes  is  a  fuzzy 
membership  function,  which  specifies  the  degree  to  which  the  node’s  input  parameter  satisfies  some 
linguistic  quantifier  associated  with  the  node.  The  D  layer  of  nodes  outputs  the  firing  strength  of  the 
fuzzy  rules,  and  the  N  layer  normalizes  the  firing  strengths.  The  consequent  layer  performs  (Sugeno- 
type)  defuzzification,  aggregated  by  a  single  weighed  sum  node  in  the  final  layer.  Fuzzy  IF-THEN  rules 
that  the  system’s  structure  is  based  on  may  be  obtained  from  human  experts  or  constructed  automatically 
based  on  the  format  of  training  data.  The  learning  rule  is  a  hybrid  of  gradient  descent  and  least  square 
estimation  of  parameters.  In  the  forward  pass  of  the  learning  algorithm,  signals  go  forward  till  layer  4 
and  the  consequent  parameters  are  identified  by  the  least  squares  estimate.  In  the  backward  pass,  the 
error  rates  propagate  backward  and  the  premise  parameters  are  updated  by  gradient  descent. 

The  neural  network  based  fuzzy  inference  system  was  trained  using  various  landing  passes  and  learned  to 
predict  future  aircraft/pilot  locations. 
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Figure  4.  Neural  Network  based  Fuzzy  Inference  System 


4  Phase  II  Objectives  and  Accomplishments 

4.1  Objectives 

The  primary  objective  was  to  buttress  LSO  decision  making  by  developing  pilot  trending  and  ship 
oscillation  recognition  decision  aids.  In  support  of  this  primary  objective  are  several  subsidiary  ones: 

•  Elicit  Important  Pilot  Trending  and  Decision  Support  Considerations 

•  Elicit  Important  Approach  Parameters  and  Similarity  Measures 

•  Elicit  Important  Pilot  Considerations  and  Similarity  Measures 

•  Develop  Pilot  Trending  and  Ship  Oscillation  Recognition  Techniques/Software 

•  Design/Implement  LSO  Interface  for  Pilot  Trending  and  Ship  Oscillation 

•  Test  Prototypes 

4.2  Data  Collection 

During  the  early  stages  of  this  Phase  II  project  we  continued  Cognitive  Task  Analysis  (CTA)  of  the 
landing  signal  officers  (LSOs)  and  began  developing  an  LSO  interface.  The  envisioned  interface  design 
was  designed  to  provide  the  LSOs  with  critical  information  in  appropriate  formats  to  better  support  their 
understanding  of  how  the  pilots  trends  and  the  oscillations  of  the  aircraft  and  ship  are  influencing  the 
current  recovery  for  a  particular  pilot  under  particular  circumstances  (i.e.,  help  them  land  the  planes 
more  safely  and  expediently  a  somewhat  contradictory  LSO  mandate).  Our  approach  was  based  on 
decision-centered  design  concepts  wherein  the  critical  decisions  and  judgments  required  of  the  LSOs 
drove  the  design  development. 
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4.2.1  Knowledge  Elicitation 

Our  primary  subject  matter  experts  for  the  CTA  were  U.S  Navy  Commander,  Frank  Pfeiffer  (ret.),  a 
former  CAG  LSO,  and  the  instructors  at  the  ISO  Training  Center  at  NAS  Oceana,  Virginia  Beach,  VA. 
CDR  Pfeiffer  served  as  a  U.S.  Naval  Pilot  and  LSO  for  many  years  and  is  currently  employed  as  a  pilot 
for  a  major  commercial  airline.  During  the  Phase  I  and  Phase  II,  we  interviewed  CDR  Pfeiffer  on 
multiple  occasions  totaling  around  100  hours.  Many  of  the  sessions  were  one  to  two  days  in  length.  In 
addition  to  sessions  with  CDR  Pfeiffer,  we  have  visited  the  LSO  School  on  about  12  different  occasions 
where  we  performed  knowledge  elicitation  on  both  the  instructors  and  students,  and  had  them  react  and 
comment  on  design  concepts  and  previous  data  analyses  we  had  conducted.  In  the  early  stages  of  the 
project,  the  LSO  School  personnel  were  additional  sources  of  data  and  in  the  latter  stages,  they  served 
primarily  for  evaluation  (providing  feedback  about  the  display)  and  testing  of  the  interface  designs, 
which  will  be  described  later  in  the  document. 

We  employed  the  Critical  Decision  Method  (CDM)  and  Knowledge  Audit  to  elicit  critical  pilot  trending 
and  aircraft  and  ship  oscillation  data,  and  used  more  informal  interviewing  methods  to  elicit  background 
information  about  the  overall  LSO  task  environment.  For  pilot  trending  we  elicited  both  the  knowledge 
relating  to  an  approach  that  is  important  for  the  LSO  to  know,  and  how  to  identify  similar  approaches. 
Foremost,  we  elicited  important  approach  parameters  and  similarity  measures  ( i.e.,  what  information 
about  the  approach  is  most  important  to  the  LSO,  and  how  is  similarity  between  approaches  defined) 

This  knowledge  was  needed  to  determine  what  should  be  displayed  to  the  LSO  and  when  it  should  be 
retrieved.  We  also  elicited  important  pilot  considerations  and  similarity  measures  (i.e.,  what  knowledge 
about  a  pilot  is  most  important  to  the  LSO,  and  how  is  similarity  between  pilots  defined,  for  purposes  of 
aiding  the  LSO). 

To  truly  employ  decision-centered  design,  we  needed  to  begin  by  understanding  how  LSOs  with 
experience  and  expertise  break  down  their  job  from  a  cognitive  perspective.  The  results  of  the  Phase  I 
CTA  revealed  that  there  were  several  key  types  of  information  that  were  critical  to  the  judgment/decision 
required  of  the  LSOs  to  perform  their  job  successfully,  and  we  used  these  as  a  springboard  for  further 
elicitation  in  the  Phase  II.  These  include: 

Deviations  of  the  aircraft  glide  path  from  the  glideslope,  dictated  by  the  basic  angle  of  the  day. 

Deviations  of  the  aircraft  speed  from  the  ideal,  based  on  its  attitude  and  the  particular  aircraft  type. 
Deviations  of  the  aircraft  line  up  from  the  centerline  of  the  landing  area  on  the  angled  deck. 

Abnormal  aircraft  configuration,  based  on  the  aircraft  type. 

It  is  important  for  us  to  stress  that  the  above  list  identifies  critical  information  requirements  of  LSOs,  but 
these  should  not  be  confused  with  decision/judgment  requirements.  The  decisions  and  judgments  will 
provide  us  contexts  within  which  these  data  are  appropriate  and  will  provide  frames  for  design  concepts. 
This  will  be  addressed  more,  later  in  the  report. 

For  each  of  the  above  points  (glide  path,  attitude,  line  up,  and  configuration)  the  CTA  in  the  Phase  II  also 
brought  out  the  important  conditions  under  which  the  landings  were  occurring  (Case  I,  II,  or  III).  Case  I 
(day)  and  II  (night)  refer  to  good  weather,  visibility,  and/or  sea  conditions.  Case  III  refers  to  poor 
weather,  visibility,  and/or  sea  conditions. 
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In  addition,  two  other  criteria  were  identified  as  important  influences  on  the  decision  making  of  the  LSO: 
the  type/model  of  aircraft  and  the  experience/competence  of  the  pilot.  The  type/model  of  aircraft  dictates 
the  characteristics  and  responsiveness  of  the  aircraft.  This  in  turn  influences  how  much  the  LSO  can  let  it 
drift  off  centerline  (the  larger  the  wingspan,  the  less  drift  that  can  be  allowed),  how  quickly  the  engine 
can  spool  up  (the  slower  the  spool  up  time,  the  further  out  the  power  call  must  occur),  etc. 

Our  knowledge  elicitation  in  Phase  II  uncovered  that  the  LSOs  have  five  categories  they  use  to  classify 
the  experience/competence  of  the  pilots:  New  Guys  (FNG),  Average  Pilots,  Top  Pilots,  Problem 
Children  (those  pilots  who  were  having  a  lot  of  difficulties),  and  Staff/COD  pilots  (those  pilots  who  did 
not  get  to  land  that  often  on  the  carrier).  During  one  of  the  data  review  sessions,  the  LSOs  said  that  they 
treat  Problem  Children  and  Staff/COD  pilots  the  same  as  FNGs,  so  we  collapsed  these  three  into  the 
FNG  category,  leaving  3  categories;  FNG,  Average,  and  Top  Pilots. 

Furthermore,  the  Phase  I  data  collection  uncovered  that  the  LSOs  consistently  visualize  the  overall 
recovery  of  aircraft  by  segments  that  are  related  to  where  the  aircraft  is  in  its  approach  pattern.  In  the 
Phase  II,  the  LSOs  helped  us  determine  that  the  critical  segments  for  the  pilot  trending  and  oscillation 
recognition  components  are  primarily  from  when  the  aircraft  is  roughly  one  mile  out  all  the  way  on  in  to 
the  deck.  While  1  mile  in  is  the  critical  area,  we  collected  data  for  the  entire  Case  I  &  II  approach  from 
where  the  aircraft  first  makes  the  pass  along  the  starboard  side  of  the  ship.  This  would  be  the  equivalent 
of  greater  than  three  miles  out  for  a  Case  III  recovery.  The  LSO  terms  for  these  segments  are: 

The  Pass 
The  Break 

The  180  (Case  I  &  II  approaches)  or  3  NM  (Nautical  Miles)  out  (Case  III). 

The  135  (Case  I  &  II  approaches)  or  2  NM  out  (Case  III). 

The  90  (Case  I  &  II  approaches)  or  1  NM  (Case  III). 

The  Start  (X).  Approximately  -  NM  from  the  ramp. 

In  the  Middle  (IM).  Approximately  NM  from  the  ramp. 

In  Close  (IC).  Approximately  1/8  NM  from  the  ramp. 

At  the  Ramp  (AR).  Right  at  the  ramp  (stem  of  the  ship) 

In/Over  the  Wires  (IW  or  OW):  The  area  where  the  arresting  wires  cross  the  landing  area.  IW  implies 
the  aircraft  has  been  trapped  while  OW  implies  it  missed  the  wires. 

Note  that  from  the  Start  (X)  on  in,  the  three  Cases  are  identical  with  the  exception  of  the  lighting 
(day/night)  and  weather  conditions  (clear,  heavy  rain,  stormy,  etc.). 

Knowledge  elicitation  only  represents  one  aspect  of  the  CTA  methodology.  Further  analysis  and  data 
representation  are  necessary  in  order  to  make  sense  of  the  information  before  the  data  is  shared,  in  this 
case,  with  the  Artificial  Intelligence  experts,  and  subsequently  fed  into  AI  models  for  development  of  a 
decision  support  tool.  The  next  section  describes  how  the  data  was  organized  into  a  useful  form  that 
could  be  used  by  AI  technicians. 

4.2.2  CTA  Representation 

Representation  of  the  data  collected  from  the  interviews  is  a  crucial  aspect  of  the  CTA  methodology.  A 
Decision  Requirements  Table  (DRT)  was  created  to  organize  the  information  that  was  elicited  in  the 
CTA  interviews  (see  appx.  T  grid  notes  and  corollary).  DRTs  are  a  popular  format  for  organizing  data. 
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especially  in  terms  of  decision  requirements  and/or  the  information  needed  to  make  the  critical  decisions. 
For  example,  we  collected  data  about  how  an  aircraft  can  deviate  from  the  glide  slope  (from  dimension 
3),  under  Case  II  conditions  (dimension  2)  within  flight  segment  IM  (dimension  1). 

We  refer  to  the  tabulation  of  information  across  dimensions  as  a  decision  requirements  table.  Figure  ? 
shows  a  sample  Decision  Requirements  Table  that  is  not  completed.  A  completed  decision  requirements 
table  for  the  LSO  is  found  in  Appendix  T.  The  decision  requirement  tables  and  cognitive  task  analyses 
from  the  Phase  I  and  Phase  II  identify  the  critical  decisions  and  judgments  around  whieh  all  design 
recommendations  are  focused  (i.e.,  decision-centered  design). 

The  following  probes  organize  the  DRT  table  illustrated  below; 

How  can  they  deviate  (e.g.,  go  high,  low,  left,  right,  etc.)? 

What  are  the  indicators  of  these  deviations  (angle  of  bank,  altitude,  etc.)? 

What  are  the  specific  cues/indicators  to  the  LSO  (how  much  right  wing  tip  is  visible,  etc.)? 

At  which  point  can  the  LSOs  discriminate  the  deviation  (e.g.,  30  ft.  off  glide  slope)? 

At  what  value  or  point  does  a  deviation  become  a  problem  (e.g.,  5  ft.  low)? 

Are  there  differences  (when  it  becomes  a  problem)  for  the  three  different  pilot  types? 

And  finally,  why  are  these  data  important  for  the  LSO?  That  is,  what  are  they  trying  to  do  that  requires 
them  to  need  this  information? 

CaseHow  Can  Deviate 
Indicators 

Specific  Cues/Indicators 
Discrimination  Ability 

When  does  Deviation  Become  a  problem?  Differences  for  Pilots/Pilot  Types? 

Glide  Slope/Glide  Path 
1200  ft.  decent  begins  Attitude/ 

SpeedLine  UpConfiguration  of  aircraftFigure  ?:  Decision  requirement  table  example 


Case 

How  Can 
Deviate 

Indicators 

SpeciHc  Cues 
and 

Indicators 

Discrimination 

Ability 

Glide 

Slope/Glide 

Path 

1200  ft.  decent 
begins 

Attitude/ 

Speed 
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Line  Up 

Configuration 
of  aircraft 

Table  1 :  Decision  requirement  table  example. 


4.2.3  Data  Sources 

In  parallel  to  the  CTA  work,  we  examined  several  other  data  sets.  These  data  sources  also  fed  into  the  AI 
models.  These  are  briefly  described  below: 

SPN-46  Radar  Data.  The  SPN-46  radar  tracks  several  aspects  of  each  approaching  aircraft  and  of  the  ship 
at  a  rate  of  approximately  15-20  samplings  per  second.  For  each  aircraft  it  collects  plane  position 
information  using  three  coordinates:  distance  from  the  ship,  horizontal  position  with  respect  to  the  ship 
(line  up),  and  altitude.  In  addition,  it  processes  some  of  these  data  to  also  provide  the  aircraft’s  closing 
speed  and  its  sink  speed.  The  SPN-46  radar  also  tracks  the  ships  pitch  and  roll  in  degrees.  Header  data 
includes  the  time-hack,  pass  number,  the  radio  channel  in  use,  and  aircraft  side  number. 

LSO  Comments/Grades.  Every  pass  is  graded  by  the  controlling/backup  (senior)  LSO.  These  grades, 
or  comments,  describe  the  aircraft’s  location  and  characteristics  for  each  segment  of  the  approach,  at 
least  from  the  start  (X)  in  to  the  wires.  An  annotated  example  of  a  LSOs  grade/comment  on  a  pass  is 
provided  in  Figure  2.  A  glossary  of  the  LSOs  grading  is  included  in  Appendix  U  Figure  2. 

Automated  Performance  and  Readiness  Training  System  (APARTS)  Program,  Data.  &  Reports. 

APARTS  is  a  program  used  by  the  LSOs  to  input  records  of  every  recovery  pass.  It  works  from  a  MS 
ACCESS  database  and  can  generate  a  variety  of  reports,  both  for  the  individual  pilot  and  the  squadrons. 

A  key  piece  of  data  that  APARTS  uses  is  the  LSOs  Grade/Comments  for  each  pass.  It  is  from  these 
inputted  APARTS  data  that  our  system  will  extract  the  information  about  a  pilots  trends  and  history. 
(Appendix  E). 

Accident  Summary  Reports.  The  U.S.  Navy  Safety  Center,  located  at  the  Norfolk  Ship  Yards,  Norfolk, 
VA  keeps  a  complete  set  of  summaries  of  all  incidents  that  occur  in  the  U.S.  Navy.  We  requested  a  set  of 
recovery  incidents  involving  fixed  wing  aircraft,  where  the  LSO  was  mentioned.  Several  hundred 
incidents  were  retrieved.  We  studied  these  for  any  possible  patterns  and  information  that  might  prove  to 
be  useful  in  this  project.  An  example  of  a  report  is  presented  below.  Note  that  the  summary  report  is 
informative,  but  also  rather  cryptic.  They  did,  however,  give  us  a  feel  for  the  range  of  incidents  that 
occurs  on  the  carriers,  beyond  the  typical  ramp  strikes  we  had  learned  about  through  other  data  sources. 
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Example  of  Naval  Safety  Center  Accident  Summary  Report 

ACFTMOD  EVENT_SERL  F014A  15753 

. EVENTSUMMARY . 

During  night  CV  bolter,  ACFT  drifted  right  &  stbd  wing  tip  impacted  2  ACFT  spotted  on  stbd  side  of 
foul  line.  ACFT  subsequently  recovered  safely.  Mishap  cause  factors:  aircrew  error  improper  landing 
technique.  Pilot  failed  to  correct  for  right  drift  at  the  ramp.  Line  up  corrections  throughout  the  approach 
were  timely  &  appropriate.  Just  after  crossing  the  ramp  ACFT  established  a  right  drift.  At  that  point 
ACFT  was  waved  off  for  being  too  high  to  land  safely.  The  pilot  transitioned  to  inside  the  cockpit  to  set 
the  attitude  &  monitor  his  instruments.  Mp  stated  he  sensed  the  right  drift  but  did  not  believe  it  to  be 
excessive  &  initiated  no  correction  prior  to  touchdown.  Controlling  personnel  factors  backup  LSO  failed 
to  make  a  timely  line  up  call  when  right  drift  was  noted  as  ACFT  crossed  the  ramp.  The  backup  LSO  is 
responsible  for  monitoring  ACFT  line  up  during  the  approach.  He  must  monitor  line  up  all  the  way  to 
touchdown  &  be  ready  to  give  a  mandatory  UHF  call  when  a  deviation  occurs,  even  if  the  ACFT  is  over 
the  wires.  The  backup  LSO  did  see  the  right  drift  at  the  ramp  but  elected  not  to  issue  a  line  up  call 
fearing  ACFT  would  touchdown  on  left  main  mount  only.  It  is  the  backup  LSO's  responsibility  to  call  for 
the  correction,  &  trust  the  pilot  to  be  aware  of  his  proximity  to  the  deck  &  make  the  correction 
appropriate  to  the  situation. 

Video  Tapes  of  Recovery  Incidents/Accidents.  The  instructors  at  the  U.S.  Navy  LSO  School  at  NAS 
Oceana,  Virginia  Beach,  VA  provided  us  with  a  video  compendium  of  carrier  recovery  incidents.  These 
were  extremely  informative,  and  sobering.  The  video  provided  us  with  a  sense  of  the  acute  time  pressure 
the  LSOs  work  within  and  the  dangerous  nature  of  the  aircraft  recovery  process. 


4.3  Pilot  Trending  Analysis 

To  perform  the  pilot  trending  research  it  was  required  that  we  get,  from  the  Navy,  data  associated  with  a 
large  number  of  carrier  landings,  for  the  full-range  of  aircraft  that  the  LSO  must  help  land.  We  tapped 
specific  data  sources  for  this  task,  which  included  SPN-46  data  (range,  bearing,  and  altitude  data  over 
time  for  the  incoming  aircraft),  and  ship  motion  data  (pitch,  roll,  heave,  and,  preferably,  velocity) 
retrieved  from  at-sea  landings.  The  most  critical  data  source  used  for  the  pilot  trending,  however,  was 
the  LSO  comments  and  grades,  which  already  existed  in  an  APARTS  data  base  for  each  pilot  (Appendix 
E). 


4.3.1  Case-Based  Reasoning  (CBR) 

We  applied  the  technique  of  Case-Based  Reasoning  (CBR)  to  address  the  Pilot  Trending  problem.  Much 
of  the  research  in  Case-Based  Reasoning  is  directed  toward  retrieving  similar  cases  and  determining 
useful  definitions  of  similarity.  For  a  pilot  trending  system,  the  cases  are  simply  previous  examples  of 
carrier  landings,  including  all  information  available  from  the  ship’s  systems,  from  which  inferences  and 
comparisons  can  be  made  using  CBR.  In  order  to  define  what  constitutes  similarity  between  approaches 
and  between  pilots  from  the  LSO  perspective,  the  LSO’s  notation  and  comments  provided  one  aspect  to 
this  representation.  These  comments  captured  an  approach’s  motion  pattern  at  a  high-level  of 
abstraction.  These  high  level  comments  then  were  used  as  a  basis  for  assessing  the  similarity  between 
two  approaches. 
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From  another  perspective,  case  representations  often  include  features  at  a  low  level,  and  features  at  a 
high  level  of  abstraction.  For  approaches,  the  low  level  features  included  the  approach  data  (range, 
bearing,  altitude),  and  the  high-level  features  included  the  LSO’s  comments.  Obviously,  since  many  of 
the  LSO’s  comments  capture  the  approach  at  a  high-level  of  abstraction,  there  was  some  redundancy  of 
information. 

CBR  uses  cases  to  record  the  experience  know-how  and  process  the  reasoning  for  retrieving  solutions 
from  such  subsequently.  A  case  is  a  contextualized  piece  of  knowledge  representing  an  experience.  It 
contains  a  past  lesson  that  is  the  content  of  the  case  and  the  context  in  which  the  lesson  can  be  used.  In 
our  project  a  case  consists  of  the  LSO  comments  of  the  flight  performance  at  different  stages  as  the 
specific  aircraft  approaches  the  aircraft  carrier  under  the  same  pilot  and  the  same  environmental 
conditions. 

The  CBR  software  delivers  two  major  objectives  and  show  its  results  on  the  display: 

•  Approaches  based  on  similar  conditions:  Retrieval  of  the  number  of  approaches  with  similar 
conditions  and  the  total  number  of  traps  per  pilot  with  this  aircraft  type. 

•  Trend  Patterns:  Retrieval  of  similar  landing  trend  patterns  from  stored  cases;  display  them 
graphically;  and  show  the  LSO  comments  of  the  closest  case. 

The  case-based  reasoning  (CBR)  for  pilot  trending  consists  of 

1)  Indexing, 

2)  Similarity  definition,  and 

3)  Retrieval  algorithm. 

We  use  CBR  to  represent  the  pilot  trending  knowledge  and  use  that  knowledge  for  LSOs  to  evaluate  the 
landing  performance;  and  to  provide  related  trending  flight  cases.  Through  CBR,  we  represent  prior 
approaches  and  the  affiliated  LSO  comments  as  cases.  Retrieval  of  similar  cases  is  then  performed  to 
provide  the  similar  cases  for  pilot  trending  analysis.  Upon  reading  the  pilot  information,  the  current 
weather  condition,  and  the  aircraft  position  (SPN-46),  the  pilot  trending  system  uses  the  case  base 
reasoning  system  to  retrieve  the  most  similar  patterns  from  previous  cases  stored  in  the  APARTS  historic 
database.  The  recent  similar  10  patterns,  and  the  current  one  are  displayed  on  the  displayed  panel  in 
graphical  format.  The  associated  LSO  comments  may  also  be  displayed. 

The  overall  pilot  trending  architecture  is  as  depicted  in  Figure  5.  The  Pilot  Trending  Analysis  box  in  the 
figure  is  handled  via  case-based  reasoning. 


19 


Stottler  Henke  Associates,  Inc. 


Final  Report 


Figure  S.  The  Architecture  Of  Pilot  Trending  System 


4.3.1. 1  Indexing 

Several  features  are  selected  in  defining  the  indexes  for  the  cases.  Indexing  facilitates  retrieval  of  similar 
cases.  The  multiple  features  selected  as  indexes  are  pilot  name,  aircraft  type,  glideslope,  lineup,  and 
day/night. 

4.3.1. 2  Similarity  Definition 

Similarity  assessment  is  the  process  of  comparing  an  incoming  flight  pattern  to  stored  approach  cases 
using  the  similarity  definition  and  indexes  to  produce  a  similarity  score.  This  is  done  progressively  as  the 
aircraft  approaches  the  aircraft  carrier.  As  more  flight  pattern  data  is  available  from  stages  of  at  the 
start(X),  in  the  middle(IM),  in  close  (IC),  to  that  of  at  the  ramp(AR),  the  similarity  assessment  is 
performed  for  each  of  the  fours  stages. 
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Figure  6.  Similarity  assessment  with  glideslope  matching 

As  shown  in  Figure  6  the  pattern  matching  degree  is  evaluated  by  comparing  the  current  glideslope 
pattern  with  that  of  the  stored  ones.  A  similar  procedure  is  applied  to  lineup  before  the  system  decides 
what  stored  case  to  be  retrieved  as  the  closest  one. 

The  above  similarity  definition  takes  the  following  into  consideration. 

i)  If  the  glideslope  difference  between  the  current  one  and  that  of  the  stored  one  is  small,  it  will 
have  a  relatively  small  matching  index  value  and  is  therefore  considered  close. 

ii)  At  each  stage  the  matching  index  takes  a  weighted  sum  on  the  mismatch  of  the  current 
stage  and  that  of  the  accumulate  mismatch  of  the  previous  stages. 

The  matching  index  is  a  direct  measure  on  how  close  the  stored  case  is  to  the  incoming  flight.  With  the 
proper  normalization  of  the  linguistic  to  numeric  conversion,  e.g., 

•  Very  High  =  0.5 

•  Medium  High  =  0.33 

•  Little  High  =  0.17 

•  OK  =  0 

•  Little  Low  = -0.17 

•  Medium  Low  =  -0.33 

•  Very  Low  =  -0.5. 
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Flieht  Path  Deviations  to  Linguistic  Conversion 

For  flight  data  that  does  not  have  associated  linguistic  data,  PADAL  has  to  determine  the  appropriate 
linguistic  conversion  from  numerical  flight  path  data.  Fuzzy  logic  is  employed  in  PADAL  to  perform 
flight  path  to  linguistic  conversion.  Fuzzy  lineup  and  glideslope  functions  are  represented  in  Figure  7. 
The  lineup  category  consists  of  7  fuzzy  sets,  ranging  from  significant  left  lineup  (_LUL_)  to  significant 
right  lineup  (_LIJR_3.  The:  glideslope  category  is  subdivided  into  7  analogous  fuzzy  sets  which  construct 
a  “very  high”  (_H_)  to  “very  low”  (1lO_)  classification  of  the  aircraft’s  glideslope.  These  fuzzy  sets 
map  directly  onto  the  comments  used  by  LSOs  to  describe  the  aircraft’s  position. 


LUL  LUL  (LUL) 


PERFECT 


LUR  LUR 


Figure  7.  Lineup  &  Glideslope  Fuzzy  Membership  Functions 


Similar  fuzzy  definitions  are  constructed  for  various  other  parameters  that  define  the  landing  trajectory. 
These  fuzzy  concepts  enable  the  system  to  classify  any  point  in  the  landing  trajectory  by  associating 
fuzzy  membership  values  with  it. 
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For  example,  a  marked  point  in  Figure  8  has  the  glideslope  deviation  from  the  nominal  glideslope  (3.5°) 
of  3.7314°-3.5°  =  0.2314°,  which  corresponds  to  the  following  glideslope  classification: 


Glideslope: 
fi_Lo_  =  0.00 

M-_h_ 

=  0.39 

=  0.00 

flH 

=  0.93 

M(lo) 

=  0.00 

fl(H) 

=  0.27 

M-PERFECT"  0.00 

This  means  that  an  aircraft  in  that  position  is  very  likely  to  be  classified  as  high  by  a  landing  signal 
officer,  somewhat  likely  to  be  classified  as  very  high  or  a  little  high,  and  extremely  unlikely  to  be 
classified  as  low. 

4.3. 1.3  Retrieval  Algorithm 

After  the  case-based  reasoner  has  determined  the  most  similar  landings,  the  retrieval  algorithm  retrieves 
these  similar  landings  from  stored  cases;  displays  them  graphically;  and  displays  the  LSO  comments  of 
the  closest  case.  Based  on  the  information  of  the  incoming  flight,  the  CBR  system  performs  and 
provides  the  following  for  display  on  the  display  panel. 

•  Total  traps  are  the  number  of  cases  that  have  the  pilot  name  index  matching  that  of  the  incoming 
pilot. 

•  Similar  conditions  are  the  number  of  cases  that  have  the  index  (pilot  name,  aircraft  type, 
squadron,  and  day/night)  matching  that  of  the  incoming  flight. 
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In  addition,  the  ten  most  recent  and  similar  stored  cases  that  match  the  current  flight  pattern  will  be 

displayed  in  four  panes,  corresponding  to  X,  IM,  IC,  and  AR  stages  respectively,  with  the  following 

features. 

i)  A  graphical  representation  of  the  past  ten  similar  flights’  trend  will  be  displayed 

ii)  Four  graphical  windows  will  be  aligned  with  the  LSO  comment  summary  for  each  of  the  four 
stages,  i.e.,  X,  IM,  IC,  and  AR. 

iii)  Each  window  contains  10  most  recent  similar  approach  data. 

iv)  In  each  window,  there  will  be  a  Cartesian  coordinate  with  the  horizontal  axis  showing  the  lineup 
and  the  vertical  axis  representing  the  glideslope. 

v)  The  trend  data  will  be  represented  as  dots  of  varying  sizes(recency).  Red  circles  may  be  used  for 
the  most  similar  ones. 


4.4  Prediction  of  Plane  Trajectory  &  Ship  Motion 

To  guide  an  aircraft  to  land  more  safely  and  smoothly  aboard  aircraft  carriers.  Landing  Signal  Officers 
(LSO)  on  board  need  to  advise  incoming  pilots  to  adjust  their  flight  patterns  continuously.  The  ability  to 
predict  how  the  aircraft  motion  trajectory  may  look  can  facilitate  LSOs  in  making  their  guiding  decision. 
Typically,  the  flight  pattern  is  carefully  observed  and  guided  when  aircraft  is  within  one  nautical  mile  (1 
NM)  from  the  landing  deck  in  open  sea.  This  corresponds  to  slightly  more  than  one  minute  in  real  flight 
time.  A  radar  system  records  all  the  trajectories  of  different  pilots  flying  various  jet  fighters.  This  data 
may  be  used  to  train  a  system  for  subsequent  prediction  purpose.  A  projection  of  2  seconds  profile  ahead 
of  the  current  flight  position  is  usually  considered  appropriate.  Another  useful  subject  that  helps  LSOs  in 
this  guiding  process  is  the  prediction  of  the  ship’s  deck  motion  in  the  forthcoming  4  seconds.  If  the  deck 
is  predicted  to  tilt  up,  an  LSO  can  advice  the  pilot  to  land  somewhat  higher  as  it  touches  the  deck. 
Misleading  prediction  may  lead  to  crash  or  waveoff.  A  reliable  prediction  algorithm  is  therefore 
essential  for  this  task. 

This  task  consisted  of  solving  a  time  series  prediction  problem  in  which  past  and  present  motion  profiles 
are  provided  to  the  prediction  system  to  predict  the  motion  in  the  next  few  seconds.  No  other 
information  was  provided  to  base  the  prediction  on,  such  as  present  engine  setting  or  wind  speed  and 
direction.  Thus  the  general  problem  was  to  take  as  input  noisy  time-series  profiles  with  a  maximum 
duration  of  about  1  minute  and  provide  a  2  second  hence  prediction  of  the  plane’s  location.  This 
problem  may  be  depicted  as  in  Figure  9. 


Figure  9.  Time  Series  Prediction 

Potential  candidates  for  solution  of  this  problem  included  statistical,  Fourier,  wavelet,  neural  networks, 
fuzzy  logic  and  other  transform  and  machine  learning  techniques. 
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The  data  set  in  the  aircraft-landing  domain  consists  of  numeric  aircraft  trajectory  curves  recorded  by  a 
radar  positioned  aboard  a  ship.  The  radar  monitors  landing  aircraft’s  lineup  (its  horizontal  distance  from 
the  center  of  the  deck)  and  glideslope  (its  approach  angle).  All  the  landing  trajectory  curves  considered 
in  the  course  of  present  study  is  subdivided  into  five  categories  based  on  the  landing  aircraft  type:  F-14A, 
F-14B,  F-18,  A-6,  and  C-2.  These  categories  provide  a  natural  way  of  subdividing  the  original 
trajectories  into  modules.  In  addition  to  the  radar-recorded  data,  new  automatically  generated  curves 
(grouped  into  modules)  were  added  to  the  system  in  order  to  determine  how  the  size  of  the  data  set 
affects  the  performance  of  the  modular  design.  These  curves  were  produced  by  a  linear  convolution  of 
the  original  curves  within  each  aircraft  category. 

Typically,  the  input  motion  profiles  can  be  clustered  into  several  loosely  coupled  categories.  This  can  be 
the  basis  of  modular  decomposition.  The  landing  motion  profiles  of  F14A,  F14B,  F18,  A6,  and  C-2,  for 
example,  are  different  though  they  share  some  common  characteristics.  Figure  9  depicts  the  input  space 
of  five  such  interconnected  clusters. 


Figure  10.  Clustered  Data  in  Input  Space 

This  modular  nature  of  the  input  data  space  was  used  to  modularize  the  design  in  the  learning  stage  by 
training  different  neural  network  based  fuzzy  inference  systems  with  respect  to  each  input  data  category. 
One  can  take  further  advantage  of  the  parallel  processing  technique  to  reduce  the  computation  time. 
Aggregation  of  these  individually  trained  modules  produced  one  generalized  module  for  prediction 
purpose.  This  generalized  module  is  expected  to  have  prediction  performance  comparable  to  that  of  the 
system  trained  with  the  traditional  non-modular  approach.  The  computation  effort  and  the  design 
complexity  are  both  expected  to  be  drastically  lower  with  the  proposed  modular  approach. 

The  neural  network  based  fuzzy  inference  system’s  ability  to  construct  an  input-output  mapping  in  a  fast 
and  efficient  manner  was  one  of  the  many  factors  that  led  to  its  selection  for  the  aircraft  prediction 
problem.  The  system  was  trained  with  a  subset  of  the  past  data  before  it  was  engaged  in  the  on-line 
prediction  task.  After  training  the  system  with  a  subset  of  the  past  profiles,  the  system  was  exposed  to 
unforeseen  approaches  and  forecast  its  profile  in  the  next  few  seconds  on-line. 
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Figure  11  shows  a  sample  aircraft  lineup  trajectory  (filtered  position),  the  trajectory  predicted  by  the 
neural  network  based  fuzzy  inference  system  (ANFIS),  and  the  trajectory  predicted  by  a  P*  order 
polynomial  extrapolation  based  upon  the  most  recent  several  seconds  of  the  trajectory  (poly  1).  The  y 
value  of  each  of  the  two  prediction  curves  at  time  t  shows  the  position  that  was  predicted  2  seconds  into 
the  future  at  time  t-2.  As  is  typical  with  time  series  prediction  algorithms,  there  is  a  tradeoff  between 
algorithms  that  respond  quickly  to  changes  in  recent  data  values  and  algorithms  that  are  tolerant  of  noise. 

We  tried  a  number  of  polynomial  prediction  algorithms  based  on  various  weightings  and  time  windows 
for  the  0*,  V\  and  2""*  derivatives  of  the  most  recent  n  seconds  of  the  trajectory.  For  each  prediction 
algorithm,  we  used  graphical  analysis  of  the  predicted  trajectories  to  understand  the  types  of  prediction 
errors  characteristic  of  each  algorithm  (undershoot,  overshoot,  lag),  and  calculated  total  prediction  error 
across  the  duration  of  each  trajectory.  We  empirically  determined  that  the  polynomial  prediction  that 
exhibited  the  lowest  error  was  a  weighted  average  of  the  current  position  and  a  linear  (F^  order) 
extrapolation  of  the  last  several  seconds  of  the  trajectory.  That  is,  predicting  the  trajectory  using  2“^* 
order  or  higher  polynomial  terms  tended  to  degrade  the  prediction.  The  neural  network  based  fuzzy 
inference  system  outperformed  this  "best"  polynomial  prediction  algorithm. 
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filtered  position 
prediction  ANFIS 
prediction  poly  1 


Figure  11.  Comparison  of  neural  network  prediction  with  polynomial  prediction  for  four  sample 

trajectories 


5  Storyboard  Evolution 

In  this  next  section  we  will  present  the  initial  storyboard  representation  and  latest  representation  of  the 
LSO  interface  produced  during  the  Phase  II.  In  this  way,  the  reader  will  be  able  to  see  how  the  display 
evolved  over  time,  and  will  help  the  readers  understand  how  the  interface  display  concepts  have  evolved 
throughout  this  project.  During  this  project  many  iterations  of  storyboards  were  created  with  the  specific 
intent  of  designing  displays  that  would  support  the  LSOs.  Because  the  interface  development  process 
involved  many  design  iterations  coupled  with  multiple  sessions  of  LSO  feedback,  it  would  be 
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cumbersome  to  present  all  versions  of  the  initial  design  concepts  in  this  report.  Also,  while  many 
features  will  be  discussed,  not  ajd  were  considered  critical  and  as  such  we  have  included  the  attached 
PADAL  CD,  Each  feature  that  we  discuss  from  the  display  will  have  a  brief  description  of  its  placement, 
why  it  is  there,  and  how  it  evolved. 
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Figure  12.  Initial  Design 

Figure  12  depicts  one  of  the  initial  concepts  for  addressing  pilot  trending  in  the  display.  This  display 
should  be  conceptualized  in  four  distinct  areas:  The  left  two-thirds  of  the  display  represents  current 
aircraft  location  with  time  and  distance  considerations,  the  top  right-third  includes  pilot  information,  the 
middle  right-third  shows  pilot  trends,  and  the  bottom  right-third  gives  the  LSO  information  as  to  what  a 
“nice  correction”  would  be. 
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Figure  13.  Pilot  Trends  and  Flight  Information 


One  of  the  later  display  designs  is  shown  in  Figure  13  above.  It  consists  of  4  main  portions  and  the 
specifications  of  each  are  outlined  as  follows:  Incoming  flight  display  with  history  and  prediction  of 
future  aircraft  position  in  the  upper  left  two-thirds,  deck  motion  history  and  prediction  by  distance  from 
ramp  in  the  bottom  left-third,  pilot  trends  by  segment  in  the  upper  right-third,  and  pilot  information  in 
the  far  bottom  right.  Everything  displayed  to  the  right  of  pilot  trends  and  information  is  for 
demonstration  purposes  only.  (Note:  we  recommend  that  all  features  can  be  turned  on  or  off  as  the  user 
wishes.) 

5.1  Potential  Sources  of  Data  to  Support  Display  Design  Recommendations 

While  the  storyboard  designs  were  being  developed  we  gave  considerable  attention  to  the  realities  of  the 
environment  and  domain  and  whether  we  would  be  able  to  acquire  the  data  necessary  to  implement  the 
display  concepts.  Figure  Y  addresses  sources  of  data  required  for  the  Graphical  Representation  (upper 
left  portion  of  the  display  area).  Figure  Z  provides  the  same  information  for  the  remaining  areas  of  the 
display. 

While  we  recognize  that  much  of  the  data  we  will  need  to  implement  in  the  displays  are  already 
available  in  one  form  or  another,  we  feel  that  we  will  need  better,  more  consistent  data  sources  to 
provide  us  some  of  the  following:a)  Today's  weather 

b)  Today's  Case  I,  II,  or  III 

c)  Past  Passes  Information  (Advanced  APARTS  database) 
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d)  Similarity  of  current  pass  based  on  similar  condition  data 

e)  Information  on  relative  importance  of  previous  recoveries 
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Figure  14.  Data  sources:  Real-time  graphs 


5.2  LSO  Interface  Design  and  Implementation  for  Pilot  Trends 

Klein  Associates  applied  decision-centered  design  to  determine  the  best  display  options  for  rapid 
decision  support.  The  LSO  domain  is  fairly  unique  in  the  requirement  for  the  LSO  to  quickly  perceive 
the  likely  pattern  of  motion  (the  approach).  Thus  the  PADAL  system  must  rapidly  convey  relevant  past 
cases  of  approaches. 

The  interface  display  is  designed  to  initially  be  a  separate  CRT  accessible  for  the  CAG/Backup  LSO  and 
is  not  being  developed  for  the  Controlling  LSO,  as  was  the  idea  in  the  Phase  I.  The  goal  is  to  take  as 
much  advantage  of  the  expertise  of  the  CAG/Backup  LSO,  to  support  this  individual's  decision  making, 
and  to  then  permit  this  individual  to  make  the  judgment  as  to  whether  or  not  some  information  should  or 
should  not  be  passed  on  to  the  Controlling  LSO  during  actual  flight  operations.- 

5.3  Evaluation 

Testing  and  evaluation  of  the  interface  concept  occurred  throughout  the  development  process.  Klein 
Associates  was  the  lead  on  this  task  and  consulted  primarily  with  Ret.  Commander  Frank  Pfieffer  and 
four  LSO  instructors  at  the  LSO  school  in  Norfolk,  VA.  Klein  Associates  and  SHAI  visited  the  LSO 
school  on  four  separate  occasions  and  talked  with  over  twenty  active  LSOs  about  the  interface  design, 
focusing  specifically  on  the  pilot  trending  and  oscillation  recognition  aids.  We  received  feedback  during 
each  of  the  four  trips  (two  separate  trials),  and  used  the  data  to  enhance  features  on  the  interface  concept. 
The  evaluation  method  was  an  informal  process  and  is  described  below. 

5.3.1  LSO  Trials 

There  were  two  major  LSO  Trials.  Each  trial  was  conducted  using  an  informal  method  of  evaluation  and 
feedback.  The  process  entailed  interviewing  highly  experienced  LSOs  about  the  design  concepts 
followed  by  LSO  reactions  to  specific  features  of  the  design.  Each  LSO  that  took  part  in  the  trials  was 
instructed  to  run  through  the  display  demonstrations  we  had  built  for  them  and  to  subsequently  provide 
feedback. 
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Klein  Associates  was  interested  in  collecting  feedback  on  many  different  aspects  of  the  display,  which 
included  factors  like  color,  size,  and  location  of  features  on  the  display.  More  importantly  though,  we 
were  interested  in  how  the  information  was  presented,  if  it  was  useful,  and  if  it  provided  support  of  their 
decision  making.  The  trials  were  not  conducted  on  an  actual  simulator,  however,  the  LSOs  were  able  to 
“put  themselves  in  the  moment”  and  comment  on  potential  environmental  factors  that  would  effect  the 
use  of  the  interface.  They  identified  factors  like  sun  glare,  proper  lighting  at  night  so  as  not  to  blind 
them,  water  on  the  screen,  and  effective  use  of  colors  so  the  features  would  stand  out  during  a  quick 
glance  of  the  screen.  In  addition  to  these  more  peripheral  display  issues,  the  first  trials,  conducted  in 
Decemeber  1999  and  February  1999  at  the  LSO  school,  primarily  concentrated  on  information  produced 
by  pilot  trending.  And  the  second  trials,  conducted  in  September  1999  at  the  LSO  school,  primarily 
demonstrated  the  results  of  the  oscillation  recognition  research  and  interface  implementation.  Following 
each  LSO  Trial  the  software  was  enhanced  based  on  LSO  feedback  and  other  results  of  the  trials. 


6  Phase  II  Prototype  System 

This  section  describes  the  design  of  the  PADAL  Phase  II  prototype.  The  system  structure  and  major 
components  of  the  PADAL  Phase  II  prototype  are  detailed  below.  The  system  has  been  deployed 
utilizing  object-oriented  design,  the  C++  computer  language  on  Pentium  class  hardware  under  MS 
Windows  operating  systems. 

6.1  Display  Design 

The  display  panel  design  is  as  shown  in  Figure  15  below. 
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Figure  15.  Display  panel  layout 

It  consists  of  5  main  portions  and  the  specifications  of  each  are  outlined  as  follows.  This  is  displayed  on 
the  upper  left  comer  of  the  display  panel  and  can  be  seen  in  Figure  16.  The  features  in  the  display  are 
described  below  in  further  detail. 


6.1.1  Incoming  graphical  display 
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Figure  16.  Incoming  flight  display 


This  is  roughly  the  2-D  equivalent  of  the  y-z  plane  of  the  3-D  trajectory  description.  A  typical  incoming 
flight  trajectory  in  3-D  is  as  shown  in  Figure  4.3.  The  center  of  the  3-D  axes  can  be  considered  as  the 
aircraft  carrier. 
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Figure  17.  Incoming  flight  trajectory  with  coordinate  axes  center  as  the  aircraft  carrier 
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In  Figure  18,  the  horizontal  axis  represents  the  ideal  glide  slope  based  on  the  Basic  Angle  in  effect  at  the 
time.  Ideally  this  angle  represents  3.5  degrees  up  and  out  from  the  deck  and  is  the  path  the  hook  should 
follow  to  touch  between  the  second  and  third  wires.  The  FRESNELL  Lens  on  board  ship  is  adjusted 
based  on  the  aircraft  type  to  match  the  particular  aircraft  hook-to-eye  value.  The  glide  slope  will  be  a 
basic  calculated  line  that  may  vary  some  with  the  specific  carrier  and  is  basic  angle  dependent. 

The  vertical  axis,  also  depicted  in  Figure  18,  represents  the  lineup  or  centerline  reference.  This  line  is 
the  actual  centerline  reference  for  the  angled  deck.  Exactly  how  the  location  of  the  aircraft  is  placed  with 
respect  to  this  line  will  be  determined  from  the  SPN-46  radar  data  and  the  dynamic  centerline. 

The  overall  frame  makes  up  about  60%  of  the  width  and  about  80%  of  the  height  of  the  display  monitor. 
It  is  situated  in  the  upper  left  comer  of  the  display  monitor.  This  feature  is  similar  to  what  the  LSOs  see 
on  the  PLAT  video,  a  screen  they  currently  use  to  determine  centerline  and  glideslope  deviations  as  the 
aircraft  approached  the  deck. 


Deviation  Circles 


Two  concentric  ellipses  represent  the  thresholds  for  reasonable  and  gross  deviations.  The  area  bounded 
by  the  smaller  ellipse  corresponds  to  reasonable  deviation  and  the  area  outside  the  larger  ellipse 
corresponds  to  gross  deviation.  Because  the  illustrations  in  this  report  are,  one  cannot  see  the  actual 
progress  from  initial  concept  to  the  current  form. 

In  the  knowledge  elicitation,  we  identified  specific  quantitative  deviations  that  LSOs  would  consider  to 
be  normal  or  gross  (see  Appendix  X,  data  specifics  piece),  based  on  each  segment  of  the  approach. 
Because  these  deviations  changed  per  each  segment,  the  size  of  the  circles  would  need  to  change  as  well. 
This  rationale  fed  into  our  initial  design  where  we  attempted  to  incorporate  dynamic  deviation  circles  as 
the  aircraft  approached  the  deck.  After  showing  this  aspect  to  many  LSOs,  the  prevailing  feedback  was 
to  leave  the  circles  static,  and  to  permit  the  LSOs  to  make  their  own  calculations  in  their  heads.  The 
LSOs  saw  circles  that  are  shrinking  and  increasing  in  size  as  more  of  a  distraction  than  as  a  support 
function. 


Ran^e  ladder 

The  range  ladder  as  shown  in  Figure  19  provides  the  x-position  information  of  the  aircraft.  The  range 
ladder  moves  up  the  centerline  as  it  approaches  the  aircraft  carrier.  The  actual  range  of  the  aircraft  is 
indicated  at  the  crosshair.  For  example,  in  Figure  19  the  x-position  of  the  aircraft  is  slightly  less  than  1 
mile  from  the  ship. 
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Figure  19.  Range  ladder 

This  tool  is  continuously  moving,  showing  the  aircraft’s  location  from  the  ramp  in  roughly  quarter-mile 
increments.  The  IC  mark  is  1/8  mile  from  AR  mark,  IM  is  1/2  mile  from  AR  mark,  X  is  3/4  mile  from 
AR  mark,  and  INM  is  1  mile  from  AR  mark.  Each  of  these  reference  marks  is  familiar  to  the  LSOs  as 
they  use  them  for  grading  segments  of  an  aircraft  approach.  It  should  be  noted  that  these  are  relative 
reference  points  and  are  not  regarded  as  absolute.  The  range  ladder  is  controlled  by  the  AR  mark  on  the 
ladder.  The  AR  mark  position  variable  is  initialized  to  be  at  the  bottom  of  the  vertical  axes  in  Figure  4.5 
and  remains  so  until  the  x-position  of  the  aircraft  is  smaller  than  1 .25  miles  and  it  will  then  begin  to 
move  dynamically  upward. 

The  position  of  the  range  ladder  has  changed  significantly  from  the  initial  design  concept  where  it  was 
seated  in  the  center  of  the  incoming  graphical  display  as  seen  in  Figure  I.2.3.4.5.6.  Many  of  the  LSOs 
found  this  to  be  distracting  and  also  commented  that  it  cluttered  the  screen.  Because  many  of  them  liked 
the  feature  so  much,  a  solution  was  to  move  it  to  the  left,  as  seen  in  Figure  4.2,  where  it  could  be  quickly 
viewed  and  unobstructed  from  other  dynamic  features.  The  range  ladder  still  utilizes  the  horizontal  axis 
as  a  reference  point  of  location. 


State  of  the  Deck 

Colored  markers  are  shown  in  the  range  ladder  to  indicate  the  state  of  the  deck.  The  position  of  the  mark 
on  the  range  ladder  is  an  indication  of  where  the  pilot  may  be  waved  off  if  the  status  of  the  deck  does  not 
change.  State  of  the  deck  was  not  considered  in  the  initial  design  concept  but  was  seen  as  an  important 
feature  to  include  after  frequent  discussions  with  the  LSOs.  In  addition,  many  of  the  latest  accidents  at 
sea  have  occurred  because  the  deck  was  not  clear  or  set  for  safe  landings.  Interestingly,  a  warning  does 
appear  on  the  current  LSO  workstation,  but  no  warning  exists  for  the  100  and  10  foot  standards  that  must 
be  met  for  clearing  a  foul  deck  and  a  deck  not  set  for  traps  (wires  are  not  retracted). 

•  A  red  mark  .#s  indicates  a  foul  deck  ( i.e.,  the  deck  is  not  ready  and  the  LSO  may  need  to 
wave  off  the  pilot  in  order  for  the  aircraft  to  clear  the  ramp  at  the  100-ft.  standard). 
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•  An  amber  mark  indicates  the  equipment  is  not  set  for  the  trap.  It  shows  at  what  point  the  LSO 
must  wave  off  the  pilot  in  order  to  clear  the  10-ft.  ramp  clearance  standard. 

•  The  amber  mark  will  turn  green  ( i.e.,  ),  to  indicate  the  deck  is  clear  and  ready  for  landing. 

These  again  are  just  reference  points  and  are  not  absolute  values.  Many  other  factors  like  aircraft  weight 
and  configuration  to  name  a  few  affect  when  and  where  an  aircraft  can  be  waved-off.  Many  of  the  LSOs 
are  very  sensitive  to  having  a  computer  determine  where  a  wave-off  can  occur  without  taking  into 
consideration  the  many  other  factors  that  would  affect  the  clearance  of  the  100  and  10  foot  standards. 
However,  given  the  recent  accidents  that  have  happened  we  still  feel  a  warning  is  necessary,  whether  it 
be  relative  or  not. 


Incoming  flisht  position 

A  small  red  solid  circle  as  shown  in  Figure  4.6  indicates  the  current  position  of  the  aircraft.  The  position 
variable  is  governed  by  the  current  reading  of  lineup  and  glideslope  deviation  (taken  from  SPN-46  radar 
data).  This  is  permanently  displayed  on  the  panel  as  a  dynamic  feature.  This  is  a  critical  piece  of 
information  since  it  is  required  before  we  can  look  at  any  trends  and/or  oscillations.  We  have  to  be  able 
to  track  where  the  aircraft  currently  is  to  be  able  to  do  any  of  these.  By  being  able  to  track  the  aircraft’s 
current  location  we  are  able  to: 

•  Identify  where  the  aircraft  has  been  in  the  past 

•  Identify  where  the  aircraft  is  with  respect  to  glideslope 

•  Identify  where  the  aircraft  is  with  respect  to  lineup  or  centerline 

•  Translate  this  information  (past  and  present)  into  pilot  trends  and  to  provide  history  for  the  entire 
pass. 


Figure  20.  Current  aircraft  position 


This  particular  feature  is  the  basis  for  all  items  listed  above.  We  intentionally  use  a  solid  circle  for 
several  reasons:  not  to  inadvertently  give  any  impression  of  knowledge  about  wing  position  (up,  down, 
etc.),  so  it  can  be  easily  recognized  on  the  screen  with  the  other  features,  and  so  that  it  shows  up  in  a  sun 
glare. 


Recent  aircraft  trajectory  profile 
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To  display  the  history  of  what  the  aircraft  has  done,  seven  gray  dots  of  diminishing  (increasing?)  sizes 
are  displayed  on  the  panel  to  represent  the  recent  7  positions  of  the  aircraft.  An  array  of  7  variables  is 
used  for  this  representation  and  each  member  of  the  array  corresponds  to: 


aircraft_profile[0] :  the  position  of  the  aircraft  at  t  -35  sec. 
aircraft_profile[l] :  the  position  of  the  aircraft  at  t  -30  sec. 
aircraft_profile[2] :  the  position  of  the  aircraft  at  t  -25  sec. 
aircraft_profile[3] :  the  position  of  the  aircraft  at  t  -20  sec. 
aircraft_profile[4] :  the  position  of  the  aircraft  at  t  -15  sec. 


aircraft jprofile[5] :  the  position  of  the  aircraft  at  t  -10  sec. 

aircraft_profile[6] :  the  position  of  the  aircraft  at  t  -5  sec.,  where  t  is  the  current  time. 


Starting  with  aircraft__profile[0],  then  aircraft_profile[l], ...,  and  ,  aircraft_profile[6],  each  is  displayed  on 
the  screen,  remaining  on  the  screen  for  about  two  seconds.  Many  variations  of  this  feature  were  talked 
about  with  the  current  design  being  the  most  comfortable  for  the  LSOs.  Many  were  concerned  with 
screen  clutter  and  the  dynamic  nature  of  the  design  being  distracting,  but  most  were  satisfied  with  the 
current  design.  The  LSOs  thought  that  the  history  could  be  picked  up  by  a  quick  glance  of  the  screen 
without  having  to  wait  long  for  the  history  to  be  displayed.  Because  it  is  difficult  to  represent  the 
dynamic  nature  of  this  in  the  report  it  is  recommended  that  the  reader  look  to  the  CD  for  a  good 
demonstration. 


6.1.2  Pilot  Trending 

Pilot  information 


The  pilot  name,  aircraft  type/model,  and  side  number  is  displayed  in  the  middle  portion  of  the  display 
panel.  It  has  the  following  format: 

Pilot  name:  String  (e.g.,  Wilcox) 

Aircraft  type:  String  (e.g.,  F14B) 

Aircraft  side  number:  (e.g.,  103) 

This  is  just  the  basic  information  about  who  is  flying,  what  is  flying,  and  which  specific  aircraft  it  is  -  all 
important  for  the  LSO  to  know.  The  pilot’s  name  conveys  a  lot  of  information  to  the  LSO  when  it  is  a 
pilot  who  is  part  of  their  squadron  or  wing  and  they  have  been  deployed  for  a  long  enough  for  the  LSOs 
to  become  familiar  with  how  the  individual  flies.  If  it  is  during  carrier  qualifications  or  it  is  a  pilot  who 
is  not  part  of  their  unit  (i.e.,  COD  pilots,  etc.)  then  less  information  is  conveyed,  other  than  it’s  a  pilot 
they  do  not  know. 

The  other  information  shown  is  already  available  to  the  LSO  but  we  feel  that  is  doesn’t  hurt  to  repeat  it. 
The  main  concern,  however,  is  that  a  pilot  may  be  “swapped”  out  for  the  day’s  flight  and  what  is  shown 
is  not  accurate.  We  have  to  assume  there  will  be  some  way  to  verify  who  the  pilot  actually  is.  One  other 
important  note  to  mention  is  that  the  pilot  name  being  displayed  has  undergone  some  serious  criticism  in 
the  past.  Two  years  ago  a  name  could  never  be  displayed  on  the  workstation  due  to  ???  But  because  of  a 
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changing  culture  and  philosophy  within  the  Navy,  and  in  particular  the  LSO  community,  the  name  being 
displayed  is  no  longer  a  negative  issue  and  is  actually  seen  as  a  helpful  feature  to  the  LSOs. 


The  Last  10  profile 

Initially  we  wanted  to  see  how  the  aircraft  was  approaching  the  carrier  and  give  trends  for  the  approach, 
based  on  how  this  particular  approach  started.  However,  the  LSOs  asked  that  we  just  show  the  pilots’ 
regular  trend  under  the  particular  conditions  for  the  current  pass  (e.g.,  night  landings  in  F/A-18B).  They 
have  also  asked  that  pilot  trends  be  flashed  up  all  at  once  as  opposed  to  sequentially  as  the  aircraft  passes 
each  segment  of  the  approach. 


Figure  21.  Example  of  Last  10 
scatter  plot  trend 
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In  figure  21,  the  right  hand  portion  of  the  display  shows  current  and  trend  data  in  the  form  of  LSO 
grades/comments.  In  our  currently  display  we  have  five  square  windows  with  cross-axis  line-up  in  a 
column  to  the  right  of  the  incoming  flight  graphical  display.  It  occupies  about  15%  of  the  total  screen 
width  as  shown  in  the  figure  below. 

The  reason  for  the  transformation  to  a  scatter  plot  was  due  to  the  feedback  received  fi'om  the  LSOs. 

Many  of  them  found  the  grading  and  comments  cumbersome  to  read,  and  many  generally  like  the  look  of 
the  scatter  plots. 

Trend  analysis  can  provide  the  LSO  with  some  very  useful  information.  However,  in  order  to  guard 
against  the  possibility  that  the  pilots  "trend",  or  average,  might  equate  to  a  zero  and  give  the  LSO  no 
trend  information  at  all,  we  have  inserted  a  scatter  plot  for  specific  sections  of  the  approach  (at  the  start, 
IM,  IC,  AR,  and  I/OW).  In  figure  21,  each  scatter  plot  displays  the  pilot's  last  ten  passes  under  similar 
conditions  (night,  case  3,  etc.).  The  current  flight  position  is  indicated  by  a  relatively  large  round  red 
dot,  the  rest  of  the  ten  recent  dots  have  varying  sizes  indicating  their  recency.  The  most  recent  one  is 
larger. 

The  scatter  plots  allow  the  LSOs  to  examine  what  a  pilot  has  done  in  their  last  ten  passes,  which  in  some 
instances  can  give  them  more  information  than  just  the  "trend."  In  addition,  it  allows  the  LSO  to  quickly 
identify  deviations  in  a  pilot's  past  performance  and  match  them  up  with  what  is  currently  happening. 

As  the  pilot  enters  each  section  of  the  pattern,  the  scatter  plot  identifies  three  approaches  from  last  ten 
passes  that  closely  match  the  current  position  of  the  aircraft.  Three  distinct  circles  appear  in  the  upper 
right  quadrant  of  the  plot  signifying  three  similar  starts.  From  this  point,  the  LSO  is  able  to  follow  these 
circles  in  each  subsequent  scatter  plot.  It  is  important  to  be  able  to  follow  these  circles  because  the 
purpose  of  the  plots  is  to  give  the  LSO  a  picture  of  how  this  pilot  looked  in  sequence  as  they  made  the 
approach  to  the  ramp  during  past  approaches. 

The  Last  10  column  is  also  shared  by  a  similar  10  feature  and  APARTS  trend  data.  The  Similar  10 
feature  was  developed  using  CBR  and  is  another  way  to  represent  pilot  trends.  Although  it  is  more 
complicated  than  the  Last  10,  similar  10  concept  is  based  on  the  initial  idea  of  how  the  aircraft  is 
approaching 

the  carrier,  and  to  give  trends  for  the  approach  based  on  how  this  particular 

approach  started.  This  is  a  useful  feature  when  a  pilot,  usually  more  experienced,  flies  many  passes.  If 
s/he  always  overshoots  the  start,  the  LSO  can  look  at  the  trend  data  on  the  scatter  plot  to  see  what  s/he 
normally  does  in  the  subsequent  segments  of  the  pass. 

The  LSOs  saw  the  APARTS  data  as  extremely  useful  because  they  use  the  tool  all  the  time.  The 
APARTS  database  is  a  tool  that  the  LSOs  use  to  enter  grading/comments  for  pilots,  and  has  its  own 
trending  system  based  on  day  or  night  landings.  However,  because  the  APARTS  database  is  not  a 
complete  database  that  is  updated,  many  of  the  LSOs  would  not  find  it  useful  on  deck. 

6.2  Interface  to  LSO  Simulator 

PADAL  has  been  designed,  implemented  and  tested  to  interface  with  the  LSO  Training  Simulator.  That 
is,  PADAL  can  receive  its  data  from  stored  information  on  disk  or  can  receive  live  pass  information  via 
ethemet  and  receive  data  from  the  LSO  Training  Simulator. 
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7  Deliverables 

The  PADAL  CD  contains: 

•  PADAL  Software,  and  the 

•  PADAL  User’s  Manual. 

8  Phase  III  and  Future  Work 

There  are  enormous  opportunities  for  the  Department  of  Defense  to  make  use  of  the  results  of  this  effort, 
both  directly  and  indirectly.  Most  directly,  the  LSO  is  the  direct  target  for  this  effort.  Furthermore,  the 
ability  to  safely  direct  the  landing  of  helicopters  is  an  important  capability  for  almost  all  Naval 
platforms.  Many  of  the  techniques  used  to  develop  the  PADAL  System  can  be  applied  to  other  platforms 
as  well. 

As  shown  in  the  following  figure,  a  portion  of  the  PADAL  Phase  I  and  Phase  II  research  will  be 
incorporated  into  the  VISUAL  system  and  thus  the  LSO  workstation  software,  running  on  the  LSO’s 
workstation  hardware.  In  particular,  portions  of  the  technology  developed  for  PADAL  has  already  been 
incorporated  into  the  VISUAL  system. 


Figure  22.  PADAL  Context 


Besides  the  contributions  PADAL  has  already  made  to  the  VISUAL  system,  there  are  important  areas 
where  Phase  III  work  could  add  to  these  contributions. 

1)  Provide  PADAL  in  a  continuous  training  environment 

Allow  for  continuous  LSO  Trials.  That  is,  a  working  PADAL  would  remain  with  the  LSO  Trainer 
(the  interface  has  been  completed  during  Phase  II  work).  This  task  would  allow  for  more  realistic 
testing  of  the  various  PADAL  components  to  better  judge  there  utility  and  thus  to  determine  which 
components  should  be  integrated  with  VISUAL.  Users  would  have  the  option  to  view  or  not  view 
PADAL.  LSO  would  provide  feedback  after  using  the  entire  trainer  and  note  whether  they  ever  used 
PADAL,  and  if  so  what  was  found  to  be  useful.  A  final  test  would  be  to  remove  PADAL  from  the 
LSO  Trainer  environment  after  many  LSOs  had  become  familiar  with  it  and  measure  reaction  to  its 
absence. 
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2)  Test  PADAL  at  sea  or  the  carrier  landing  practice  field 

Conduct  a  LSO  Trail  on  an  aircraft  carrier  or  at  the  carrier  landing  practice  field,  in  a  non-obstructive 
manner. 

3)  Cone  for  showing  prediction  of  ship 

Continue  to  develop  the  ship-motion-prediction  algorithm  to  provide  a  cone  of  prediction.  It  is 
difficult  to  predict  the  four  second  hence  deck  location  due  to  the  limited  information  provided  for 
prediction,  however,  the  prediction  could  be  enhanced  by  providing  a  cone  of  prediction  as  shown 
below.  The  thick 


black  line  is  the  past  motion  with  the  present  position  shown  at  the  rightmost  side  of  the  line;  the 
prediction  cone  shows  the  range  of  possible  locations  the  deck  may  take  during  the  next  4  seconds. 
The  cone  would  be  much  more  descriptive  than  the  example  shown,  for  it  could  be  color  shaded  to 
represent  the  likelihood  of  the  deck  being  in  a  certain  part  of  the  cone. 

4)  Improve  plane-prediction  algorithm  (with  more  data) 

Continue  to  develop  the  plan-prediction  algorithm.  This  development  is  critically  dependent  on 
more  landing  data.  With  more  landings  the  present  algorithms  can  be  further  tested  and  enhanced. 
In  addition,  the  prediction  can  be  modified  from  a  point  prediction  to  a  range  prediction  where  the 
future  location  of  the  plane  is  represented  to  show  a  region  where  the  plane  will  most  likely  to  be. 
This  is  similar  to  the  proposed  modification  to  the  ship-motion  prediction. 

5)  Offline  version  with  playback  capability 

LSOs  have  requested  a  playback  capability  for  debriefing,  review,  and  instructional  purposes. 
Requested  features  in  the  playback  version  include  pause,  random  playback  at  any  stage,  and 
slow/fast  play.  It  is  also  preferred  that  the  offline  version  is  network  accessible  with  password 
protection  in  different  levels. 

This  capability  has  been  requested  at  all  the  LSO  Trials  and  during  the  PADAL  Q&A  session  at  the 
OAG  meeting.  This  playback  capability  could  be  first  implemented  via  the  LSO  Trainer,  such  that 
whenever  the  system  is  attached  to  the  LSO  Trainer  all  passes  would  be  recorded  for  later  playback. 

6)  Ramp  motion  alarm 

LSO  requested  an  alarm  if  the  ramp  motion  exceeds  8  ft/second;  also  alarm  if  ramp  motion  ‘changes 
significantly’.  The  specifics  of  change  significantly  would  need  to  be  elicited  from  the  LSO 
community. 

7)  Adaptively  learn  ranges  corresponding  to  LSO  shorthand  comments 

The  ranges  of  deviation  corresponding  to  the  various  LSO  shorthand  comments  where  determined 
from  knowledge  elicitation  from  various  LSOs,  and  implemented  using  fuzzy  system  approach.  The 
actual  ranges  may  be  different  in  actuality  than  as  described  on  diy  land  and  the  ranges  may  change 
over  time  and  under  different  conditions. 

PADAL  could  be  modified  to  adaptively  learn  and  update  the  fuzzy  system  for  determining  the 
appropriate  ranges  that  correspond  to  the  LSO  shorthand  comments.  The  information  used  to  learn 
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the  ranges  could  also  be  used  to  graphically  depict  via  a  scatter  plot  the  actual  aircraft  location  versus 
LSO  comments.  This  could  be  used  as  a  training  and  review  tool. 

8)  Unified  Aircraft  Glideslope,  lineup  and  ship  deck  pitch  display  with  predictions 

In  the  figure  below,  the  gray  areas  are  present  to  indicate  a  different  scale.  These  indicate  2  miles, 
while  the  entire  remainder  of  the  display  concentrates  on  that  last  mile.  As  we  have  been  doing,  we 
would  still  provide  predictions  for  the  a/c  glide  slope,  lineup  and  the  deck  motion  (shown  in  red  in 
the  figure).  So  we  get  history,  actual,  and  predicted  for  these  three.  The  focus  would  be  to  take  this 
initial  integrated  display  concept  and  hone  it  to  create  one  integrated  display  that  provides  valuable 


information  and  can  be  quickly  comprehended. 


In  addition  to  the  LSO  specific  applications,  the  advanced  motion  case  display  technology  developed 
for  LSOs  would  be  useful  in  domains  where  past  cases  of  motion  patterns  can  be  retrieved  and  when 
this  historic  data  must  be  quickly  conveyed  to  the  user.  One  example  is  threat  assessment  in 
complicated  tactical  scenarios  (especially  those  related  to  defense)  such  as  those  faced  by  the  E2C 
officer,  the  AAWC  (Anti-Air  Warfare  Coordinator)  in  the  CIC,  and  the  Weapons  Director  aboard  an 
AW  ACS.  SHAI  has  contacts  in  all  of  these  fields. 

9  Conclusions 

This  report  has  summarized  the  tasks  and  results  of  this  Phase  II SBIR  project.  The  project  utilized 
artificial  intelligence  and  cognitive  task  analysis  to  develop  a  LSO  driven  decision  support  tool. 

The  project  determined  the  significant  aircraft  approach  parameters  and  similarity  measures  and 
important  pilot  considerations  and  similarity  measures.  From  this  information  the  project  developed 
pilot  trending  techniques  and  software  using  case-based  reasoning  and  combinations  of  other  AI 
techniques.  In  addition,  in  conjunction  with  many  LSOs,  the  project  determined  the  best  display  options 
and  most  appropriate  display  logic  for  the  information  produced  by  the  pilot  trending  and  oscillation 
recognition  modules,  and  designed  and  implemented  the  resulting  LSO  interface. 
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Appendix  B:  Corollary  Grid  Notes 


The  numbers  can  be  found  in  certain  cells  within  the  DRTs  in  Appendix  A.  To  save  room  in  the  tables 
we  have  decided  to  add  a  description  in  this  section  that  corresponds  to  those  numbers. 


Number 


1 

2 

3_ 

4 


5 


6 


]_ 

8 


9 


10 

11 


12 


13 


14 


15 


Description 

Kts:  Nautical  Miles  Per  Hour  _ 

CCA:  Carrier  Contiolled  Approach _ 

AOA:  Angle  of  Attack  _ 

CCA  calls:  If  not  in  EMCON  and  if  aircraft  is  getting  voice  command  form  CATC  controller.  Also, 
LSO  may  not  be  on  pilot’s  frequency  at  this  point  since  Case  III  are  DUAL  FREQUENCY.  For  Case 
III,  a  different  frequency  is  set  for  every  other  aircraft:  aircraft  101  is  on  freq.  A,  aircraft  102  is  on 

freq.  B,  aircraft  103  is  on  freq.  A,  etc... _  ' _ 

Approach  light  on  and  steady  means  that  gear  is  down.  Approach  light  steady  means  that  hook  is 
down.  Approach  light  flashing  means  hook  is  up.  However,  at  night,  if  one  of  the  three  approach 
lights  (  red  =  fast,  amber  =  on  speed,  green  =  slow)  are  burned  out,  and  this  is  the  light  that  should  be 
illuminated  given  the  aircraft  speed  at  the  time,  then  the  appearance  will  be  that  the  gear  are  up  (e.g., 
no  approach  light).  In  this  case  the  LSO  will  ask  the  pilot  to  “show  me  as  FAST”,  “show  me  as 
SLOW”,  or  “show  me  as  an  APPROACH  light.”  The  pilot  will  momentarily  dip  his  nose  or  pull  up 
slightly  in  order  to  get  the  AOA  to  change  enough  to  illuminate  one  of  the  other  two  lights  to  confirm 

gear  down.  If  no  lights  working,  a  W/0  is  required  with  a  fly  by  to  confirm  gear  down  or  not. _ 

Discrimination  ability:  Case  I  and  II  -  Line  up  at  the  135E  position  (e.g.  rate  of  turn  which  results  in  a 
line  up  outcome)  is  a  function  of  perceiving  rate  of  turn,  ship’s  track,  relative  wind  and  turn  rate 
corrections  by  the  pilot.  At  this  point  little  attention  and  little  LSO  input  can  be  made.  Discrimination 

is  gross  and  difficult  to  quantify. _ _ 

Fpm:  feet  per  mile _ _ 

At  1  NM,  if  visibility  permits,  pilots  will  begin  seeing  the  ball  and  visual  line  up  cues.  Most  will 
include  this  information  in  their  scan  of  instruments.  This  is  a  critical  time  for  pilots  in  that  most  will 
perceive  themselves  as  high  on  the  G/S  at  this  “first  look”  at  the  ship  and  tend  to  settle  below  the  G/S. 

Difficult  for  the  pilot  to  see  anything  at  1  NM.  _ 

At  1  NM  last  aircraft  should  have  trapped  and  next  aircraft  will  be  on  theJLSO’s  frequency  and  HUD 

display. _  . 

The  difference  bet^veen  discrimination  ability  is  a  function  of  perceptive  ability  primarily  at  night. 

Case  I  and  II  for  Attitude/Speed:  At  the  90E,  pilot  will  see  the  result  of  his/her  approach  turn  from 
ISOEto  the  90E  and  will  make  most  of  the  major  roll  corrections  from  the  90E  to  X.  These  changes  in 

AOB  require  A/S  changes  to  maintain  on  speed.  These  changes  are  difficult  to  accurately  see. _ 

Case  III  for  Attitude/Speed:  In  addition  to  simply  fast  or  slow,  and  more  critical,  is  the  circumstances 
of  a  pilot  who  has  not  been  able  to  resolve  his  speed  after  2  miles  of  descent  on  the  G/S.  Average 
performance  is  consistently  a  little  fast,  poor  is  >  8  kts  fast,  consistently  slow  or  unstable  slow  to  on 

speed  to  fast  to  on  speed  to  slow  equals  lots  of  problems. _ 

Case  I  and  II  for  Line  up  at  90E:  A  turn  requiring  >  45E  AOB  is  difficult  at  best.  However,  a  shallow 
turn  of  <  15E  AOB  will  necessitate  a  rapid  increase  to  over  45E  AOB  to  stop  the  aircraft  on  the  C/L  at 

the  X  and  is  therefore  a  more  critical  error/correction  association. _ _ ^ _ 

G/S  &G/P  all  cases:  Although  a  nose  down  (ND)  action  by  the  pilot  would  appear  to  be  an  A/S 
correction,  IM  -  AR  is  a  quick  G/P  correction  to  increase  ROD.  In  fact,  if  s  quicker  than  waiting  for  a 
power  reduction  to  affect  sink  rate.  Improper,  but  not  uncommon,  it  is  somewhat  acceptable  but  can 

have  disastrous  consequences  if  not  properly  executed  and  corrected. _ 

All  specific  cues  are  amplified  by  aircrafts  relative  size  and  peripheral  environment  from  IM  -  AR. 


Stottler  Henke  Associates,  Inc. 


Final  Report 


1 6  However,  very  rare  since  aircraft  are  in  level  flight  prior  to  G/S  intercept  at  200  ft.  This  is  basic 
_ instrument  flying  at  this  point. _ 


Additional  Notes 


<  Case  I  and  II  use  primary  frequency  A.  Case  III  uses  dual  frequency. 


RAMP 


I 
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The  controller  is  talking  to  101/A  and  103/A.  When  ball  is  called,  controller  stops  talking  to  103/A  for 
about  20  seconds,  and  the  LSO  does  not  stop  talking.  Before  this,  LSO  gets  G/S  information  from  the 
controller’s  calls. 

<  The  new  guy  is  easiest  to  control  because  of  heightened  expectation  of  problem. 

<  For  estimation  of  airspeed,  actual  perception  is  the  degree  of  attitude,  which  translates  to  kts. 

<  HUD  not  available  @  2  NM.  Only  when  aircraft  is  IC  - 1  NM  (unless  1“  aircraft  is  in). 

<  At  the  90E,  G/S,  attitude  and  L/U  are  starting  to  get  more  and  more  “mashed”  together. 

<  ND  will  cause  a  settle  and  this  is  want  we  want  to  avoid. 

<  Notice  that  as  pilots  get  closer  to  the  ramp,  the  Top  pilot  has  a  possible  -  probable  problem  IC-AR. 
The  reason  for  this  is  because  LSO  doesn’t  expect  Top  pilot  to  be  out  of  parameters  here.  If  he  is, 
he  may  have  more  problems.  If  Top  guy  has  problems  here,  he  becomes  more  like  the  FNG. 

<  No  approach  light  check  required  during  daytime  because  LSO  can  get  a  visual. 


Stottler  Henke  Associates,  Inc, 


Final  Report 


Appendix  C:  SPN46  Track  Data 


Header  Contains: 


Label 

Pass  Number 

Channel 

Acft  Side  No. 

RECORD 

00056 

A 

105 

15:41:14 

DATA  CONTAINS: 


X 

position 

(ft) 

Y 

position 

(ft) 

Z 

position 

(ft) 

Ship’s 

pitch 

(deg) 

Ship’s 

roll 

(deg) 

Closing 

speed 

(ft/sec) 

Sink 

speed 

(ft/sec) 

00:44.4500 

2866 

-37.50 

220 

.27466 

.70862 

-218 

-6.40625 
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Appendix  D:  Comment  Examples  and  Explanations 


LSO  COMMENTS  (Grade) 

INTERPRETATION 

(LOOSX)  /CBIM  HIC  (HCDAR) 

jSlightly  Low,  Over  Shot  at  the  Start.  Flew  up  through  the  Glide  Path  in  th 
iSlightly  high,  come  down  at  the  Ramp. 

SRD.OSX  HSLOIM-IC  PNU.CDAR 

jStopped  Rate  of  Descent  on  overshooting  the  start.  High  and  slow  from  i 
iPulled  nose  up  on  coming  down  at  the  ramp. 

OC(S.X)  /iM  HIC  HCDAR 

jOver  control  and  a  little  settle  on  the  start.  Fly  up  through  glide  path  in  th 
High  and  come  down  at  the  ramp. 

(HX)  TMP.CDIM  HIC  (PNU.CDAR) 

1 

Slightly  high  start.  Too  much  power  on  come  down  in  the  middle.  High  ir 
[nose  up  on  come  down  at  the  ramp. 

(LOX)  /IM  HIC  \AR 

Slightly  low  at  the  start.  Fly  up  through  the  glide  path  in  the  middle.  High 
through  the  glide  path  at  the  ramp. 

S.X  LOIM  NEP.COIC  LOAR 

Settled  on  the  start.  Low  in  the  middle.  Not  enough  power  on  come  on  ir 

(HX)  OCNEP.CDIM  HFIC-AR 

Slightly  high  at  the  start.  Over  controls  and  not  enough  power  on  coming 
and  fast  in  close  through  the  ramp. 

(HWUX)  OC(SIM)  (HFIC)  (HCDAR) 

jA  little  high  and  wrapped  up  at  the  start.  Over  controls  and  settles  a  little 
fand  fast  in  close.  Slightly  high  and  comes  down  at  the  ramp. 

LOLURX  NEP.CBIM  LOIC  LOBAR 

Low  and  lined  up  right  at  the  start.  Not  enough  power  on  come  down  in  tl 
Low  and  flat  at  the  ramp. 

(HX-IM)  (TMP.CDIC)  (HAR) 

|A  little  high  from  the  start  to  the  middle.  A  little  too  much  power  on  come 
high  at  the  ramp. 

(LOX)  NEP.COIM  LOIC  LOBAR 

|A  little  low  at  the  start.  Not  enough  power  on  come  on  in  the  middle.  Lov 
ithe  ramp. 

HAW 

High  all  the  way 

(LOX)  NEP.COIM  S.RUFWIC  LOAR 

A  little  low  at  the  start.  Not  enough  power  on  come  on  in  the  middle.  Set 
Low  at  the  ramp. 

SRD.(LOX)  /IM  (TMP.CDIC)  (HCDAR) 

jStopped  rate  of  descent  on  slight  low  at  the  start.  Flies  up  through  glide  [ 
(too  much  power  on  come  down  in  close.  Slightly  high  and  comes  down  £ 

(LOX)  /IM  _HIC_  HCDAR 

1 A  little  low  at  the  start.  Flies  up  through  the  glide  path  in  the  middle.  V/e 
comes  down  at  the  ramp. 

(LOX)/IM  HDRIC  (\.LUAR) 

jA  little  low  at  the  start.  Flies  up  through  the  glide  path  in  the  middle.  Higt 
[slightly  flies  down  through  the  glide  path  on  lineup  at  the  ramp. 

TMP(HX)  HIM-IC  (HCDAR) 

iToo  much  power  and  slightly  high  at  the  start.  High  in  the  middle  through 
pomes  down  at  the  ramp. 

(H.OSX)  NEP.CBIM  \IC  LOAR 

|A  little  high  on  overshooting  the  start.  Not  enough  power  on  come  back  ( 
[down  through  glide  path  in  close.  Low  at  the  ramp. 

OSX  TMP.CBIM  HIC  (HCD.LUAR) 

pver  shoots  the  start.  Too  much  power  on  come  back  in  the  middle.  Higl 
pomes  down  on  lineup  at  the  ramp 

S.LULX  (/IM)  (HIC-AR) 

Settles  on  lineup  left  at  the  start.  Flies  up  a  little  through  the  glide  path  in 
Iclose  to  ramp. 

(OSX)  (TMP.CBIM)  (HIC)  (HCDAR) 

Slightly  overshoots  the  start.  A  little  too  much  power  on  come  back  (lineu 
ligh  in  close.  Slightly  high  and  comes  down  at  the  ramp 

HX-IM  NEP.CDIC  \AR 

[High  from  the  start  to  the  middle.  Not  enough  power  on  come  down  in  ck 
'glide  path  at  the  ramp. 

OC(TMRDX)  (HIM)  (HCDIC-AR) 

pver  controls  and  has  a  little  too  much  rate  of  descent  at  the  start.  A  bit  1 
jhigh  and  comes  down  from  in  close  to  the  ramp. 

(HAA.X)  (TMP.CDIM)  (HIC)  (VAR) 

jSlightly  high,  angling  approach  at  the  start.  A  little  too  much  power  on  co 
!a  little  high  in  close.  Flies  down  through  glide  path  at  the  ramp. 

(LOX)  TMP.COIM  /IC  HCD.LUAR 

|A  little  low  at  the  start.  Too  much  power  on  come  on  in  the  middle.  Flies 
[inclose.  High,  comes  down  on  lineup  at  the  ramp. 

A  little  low  from  the  start  to  the  middle.  Flies  up  through  glide  path  in  clos 

(NESA)  (HIM)  HIC  (VAR) 

[Not  quite  enough  straight  away.  A  little  high  in  the  middle.  High  in  close.  Flie 
Iglide  path  at  the  ramp. 
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nrk  _ r  ««««  _  ^ 


LIG  H.OSX-IM  (NHP.CDIC)  (VAR) 

(ToX-IM)  NEP.DLIC  L0LULAR“ 
LIG  LO90-X  (/IM)  (\IC)  (LOAR) 
plX)  OC(TMP.CDIM)  \IC  LOBAR 


(LIG)  TMP(Wux)  HIM-IC  HCDAR 

SX)  (\IM)  (LOBIC-AR) 

(LOX)  /IM  HCDIC 
^X)  (TMP.CuiM)  (HCDIC-AR) 


m  lamp. _  L/n  Cl 

ptopped  rate  of  descent  on  the  start  Hit^h  in  thf  xr  1 - ; - - - 

U  n  through  the  glide  path  at  the  ^ ‘ 

groove  (wings  level  &  in  the  groove  at  7  ■>  j - 

start  to  the  middle.  Not  quite  enouoh  nnivAr  „  ^ 2  seconds  or  more  out).  H 

Uhatth^e  ramp  power  on  come  down  in  close.  Flies  a  litt 


Tx  TMP.DLIM  _H_LULIC  HFAR 

5c(nepim7^c1icda^ 


A  httle  too  much  power  at  the  start.  Little  hi^h  in  the  midrtl*:.  t - 1: - - 

High  and  flat  at  ramp.  ‘o"  m  tne  middle.  Too  much  power 


Stottler  Henke  Associates,  Inc. 


Final  Report 


NESA  TMRDOT  \X  LOIM  ND/IC  FEAR 

Not  enough  straight  away.  Too  much  rate  of  descent  out  of  the  turn.  Fly  down 
start.  Low  in  the  middle.  Nose  down,  fly  up  through  the  glide  path  in  close.  Fa 

NEPaOX)  LOSLOIM 

Not  enough  power  and  slightly  low  at  the  start  'Very  low  and  slow  in  the  middle 

LIG  (LOX)  /IM  OCHIC  VAR  LL 

[Long  in  the  groove.  Slightly  low  at  the  start.  Fly  up  through  the  glide  path  in  tl 
iiish  in  close.  Flv  down  through  the  glide  path  at  the  ramp.  Land  Left. 

SRDX  HIM  DEC.CDIC  VAR 

1  Stopped  rate  of  descent  at  the  start.  High  in  the  middle  Decelerate  on  coming  d 
■throueh  elide  slope  at  the  ramp. 

(NESA)  (HIM)  NEP.CDIC  LOBAR 

Not  quite  enough  straight  away.  A  little  high  in  the  middle.  Not  enough  power 
and  fiat  at  the  ramp. 

(LIG)  (H)OSX  TMP.CBIM  HIC-AR 

A  little  long  in  the  groove.  A  little  high  and  then  overshot  the  start.  Too  much  p 
middle.  High  from  in  close  to  at  the  ramp. 

LOOT  (LOLULX-IM)  (/IC)  (HCDAR) 

Left  out  of  the  turn.  A  little  Low  and  lined  up  a  little  left  from  the  start  to  in  the 
{through  the  glide  path  in  close.  A  little  high,  come  down  at  the  ramp. _ 

(TMP.X)  (HIM-IC) 

iA  little  too  much  power  on  the  start.  A  little  high  from  in  the  middle  to  in  close 

(HX)  _NEPCDIM_  LOCHLUIC-AR 

Little  high  at  the  start.  Not  nearly  enough  power,  come  down  in  the  middle.  Lo 
Iclose  to  at  the  ramp. 

SRD.X  HIM-IC  HCDAR 

Stopped  rate  of  descent  on  start.  High  from  in  the  middle  to  in  close.  High,  cor. 

(SX)  OC(LOIM)  HIC-AR 

Settle  a  little  at  the  start.  Over  control,  a  little  low  in  the  middle.  High  from  in  c 

(TMRDX)  NEPLOIM  LOIC 

1a  little  too  much  rate  of  descent  at  the  start.  Not  nearly  enough  power,  low  in  tl 

LOX  (/IM)  (HIC)  (HCDAR) 

jLow  at  the  start.  Fly  a  little  up  through  glide  path  in  the  middle.  A  little  high  in 
down  at  the  ramp. 

(LOOSX)  /IM  HCDIC  PNU.CDAR 

A  little  low  and  overshoot  the  start.  Fly  a  little  up  through  the  glide  path  in  the  i 
close.  Pull  nose  up  on  come  down  at  the  ramp. 

LOOSX/.CBIM  HIC  VAR 

Low  and  overshoot  the  start.  Fly  up  through  the  glide  path  and  come  back  in  tb 
down  through  glide  path  at  the  ramp. 

TMP.OSX  HIM  PNU.HIC  SLO.CDAR 

P'00  much  power  on  overshooting  the  start.  High  in  the  middle.  Pull  nose  up  oi 
down  at  the  ramp. 

(/.OSX)  TMP.CBIM  HIC  (HCDAR) 

[ 

Fly  a  little  up  through  glide  path  on  overshooting  the  start.  Too  much  power  on 
High  in  close.  A  little  high,  come  down  at  the  ramp. 

OSX  TMP.DLIM  HIC-AR 

iOvershoot  the  start.  Too  much  power  on  drift  left  in  the  middle.  High  from  in  c 

(HWUX)  TMP.LUIM  HIC-AR  CDTL 

A  little  high,  wrapped  up  at  the  start.  Too  much  power  on  line  up  in  the  middle, 
ramp.  Coming  down  to  land. 

(LO)OSX  /.CBIM  HDLIC  (V.LUAR) 

|A  little  low,  and  overshoot  the  start.  Fly  up  through  glide  path  on  come  back  in : 
tenter.  A  little  flv  down  though  glide  path  on  lineup  at  ramp. 

HOSX  TMP.DLIM  HLULIC  HCD.LUAR 

High,  overshoot  start.  Too  much  power  on  drift  left  in  middle.  High,  lineup  lef 
on  lineup  at  the  ramp. 

SRD.(OSX)  HDLIM  HLULIC  HCD.LUAR 

Stopped  rate  of  descent.  Slightly  overshoot  the  start.  High,  drift  left  in  the  midc 
Hieh.  come  down  on  lineup  at  the  ramp. 

OC(S.X)  (/IM)  (DR.CDIC)  (V.LUAR) 

'Over  control  and  slight  settle  on  the  start.  Fly  up  through  glide  path  a  little  in  th( 
come  down  in  center.  Fly  a  little  down  through  glide  path  on  lineup  at  the  ramp 

(HX)  NEP.CDLM  LOBIC-AR 

lA  little  high  at  the  start.  Not  enough  power  on  come  down  in  the  middle.  Low 
the  ramp. 

(LOX)  (/DRIM)  (HIC)  (CD.LUAR) 

jA  little  low  at  the  stan.  Fly  a  little  up  through  the  glide  path  and  drift  right  in  th 
close.  Slisht  come  down  on  lineup  at  the  ramp. 

(LO)OSX  (TMP.CBIM)  (HIC)  (HCDAR) 

jA  little  low  and  overshoot  the  start.  A  little  too  much  power  on  come  back  in  th 
tlose.  A  little  high,  come  down  at  the  ramp. 

S.X  LOIM  /IC  HCDAR 

Settle  at  the  start.  Low  in  the  middle.  Climb  up  through  the  glide  path  in  close, 
ramp. 

(LO)OSX  LURIM  (NEP.CDIC)  (CD.LUAR) 

'A  little  low  and  overshoot  the  start.  Lineup  right  in  the  middle.  Not  quite  enou; 
m  close.  A  slight  come  down  on  lineup  at  the  ramp. 

(LIG)  (NEPIC)  (LOAR) 

A  bit  Long  in  the  groove.  Not  quite  enough  power  in  close.  A  little  low  at  the  r 

(OSX)  (H.CBIM)  (NEP.CDIC)  (LOAR) 

Slightly  overshoot  the  start.  A  little  high  on  come  back  in  middle.  Not  quite  en 
close.  A  little  low  at  the  ramp. 

(LOOSX)  OCNEP.CBIM  HIC  HCDAR 

Slightly  low  and  overshoot  the  start.  Over  control,  not  enough  power  on  come  \ 
'close.  High,  come  down  at  ramp. 
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HOOT-X  (HIM)  (HCDIC-AR) 

High  out  of  the  turn  to  the  start.  Slightly  high  in  the  middle.  Slightly  high  ar 
the  ramp. 

LOOSX  /.CBIM  HBIC-AR 

Low  and  over  shoot  the  start.  Fly  up  through  glide  path  on  come  back  in  mid 
to  the  ramp. 

(LURX)  (TMPIM)  (HIC)  (HCDAR) 

A  little  lined  up  right  at  the  start.  A  little  too  much  power  in  the  middle.  A  li 
come  down  at  the  ramp. 

SRDIM  HIC-AR 

Stopped  rate  of  descent  in  the  middle.  Hi.eh  from  in  close  to  at  the  ramp. 

HVVUX  TMP.CDIM  HIC  HBAR 

High  and  wrapped  up  at  the  start.  Too  much  power  on  dome  down  in  the  mi 
flat  at  the  ramp. 

(LURX)  (NEP.LUIM)  (SIC)  (LOBAR) 

A  little  lined  up  right  at  the  start.  Not  quite  enough  power  on  line  up  in  the  m 
little  low  and  flat  at  the  ramp. 

— - 

^^^«®nHc5ARr 


;^&iS£25!?ssU^' 
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1  Mile  - - 

HighIftI  - 

LowTfi) 

Fast  rnnipM  - 

miFfnmph)  ^  ~ 

LefTift)  - 

LRightTm  - 

(near  the 

^ 

T^v  (ft)  — — 
Fast  (nmph)  ' 

■  Siovvto^^j^h)  ^  - 

LefUftP^ - - — 

pRightTflV  - 

Middle  “riU”\ 
I  High  ( ft)  '  ~ - ^ 

I  Low  ^ 

LFpTnmnM  '  - 

Plowtnniph^  ~ 

I  LeftTfp^  ’ 

I  ^ightTfi)  '  ~ - — 

[l/8  Mile  ( In  ringp  “Tr'”r 
I  HighTfi)  - ^ - - 

I  Lowlft) 

l-FastTnmp' - 

Li>low(nmph)  ^  - 

I  LefUffl^  - - 

iRightlft)  ~ - - - 

LA^e  Ramp  ('•‘ar’’)  " 

I  High  (ft)  - 

"Low  (ft)  ■  - - - - 


Fast  (nmph) 
SlovvTnm^iTi 
Left  (ft) 

RightTlP 

^B=  Angle  ofBanJT 


no  correction  Sf  k  !’  ‘correction 

needed 


—  FJo  Limit 
_  25 

71 

I 

~f>37l 

>37,5 

'Is  ^ — 

_~123  ~~ 

1  (Case  1  &  2) 
2,5  (Case3i 

Ij7j~  - - 

lyis 


Gross  Deviation 


ROD  =  Rate 


pitch _ 

_>L5°  pitch _ 

2-5  or  >1.5°  APR 
TJ  or  >1.5°  APR 
■  of  Descent 


- -  No  limit 

_  Tso 

f  10  — - 

MO  - - — 

7  fH  ^  — 

r>75~  • — - 

50  - - 

jlT  - - - 

T 10  (Case  1  &2) 

—  ^  (Case  3) 

71 - 

\tT~  - - r 

|ir  — 

40  ^  - - 

fid - 

fur 

I  50  Case  1)  ' 

— pQ-*-  (Case  2  &  31 
I  50  Case  1) 

— pO+(Case2<S:3) 

"[lo  ^ ^ — 

- j  or  -^5  chanpp  -  • 

_.|  5  or  >  5  chancre 
-i25or>5°AOB 
-1 25ar  >5°  AOB 

lo  or  >2uutpm  ROD  off 
ideal _ 

~^r  >^uofpm  ROD  off 
ideal 

ripitch  ' 

c>3°  pitch 

7^  or  >3°  APR  ~ 


.  No  limit 
100 
lo 
fld~ 

7i50 

llso 


20  (Case  1  &  2) 
10  (Case  3i 
MO  - 

Iso  — 

Iso  — 


[  >6°  pitch 
>6°  pitch 

30 

17“ 
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riables 


i*^gi  speed 
rate 
P 

slope 

■ 

e  Start) 
the 
e) 

Close) 

the 


Jit' 

ft-  he2„'’n°v" ““ft 
fl/sec  dm  of 'he  aircraft 

^sec  I  d2/dt,’the  raeo°f change  o°f 2  ^fh 
*•  I  This  is  the  deviation  of  th/ 

<-e*.  «>= 

approximately  1/2-  va  f  I 

approximate,, 

^j^ZZ'ZptZZnhe  fwp'T'’  ='”'P  I 


